2023-01-15T13:28:36.271 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2023-01-15T13:28:36.381 INFO:teuthology.lock.ops:Start node 'smithi150.front.sepia.ceph.com' reimaging 2023-01-15T13:28:36.381 INFO:teuthology.lock.ops:Updating [smithi150.front.sepia.ceph.com]: reset os type and version on server 2023-01-15T13:28:36.382 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2023-01-15T13:28:36.397 INFO:teuthology.lock.ops:Node 'smithi150.front.sepia.ceph.com' reimaging is complete 2023-01-15T13:28:36.397 INFO:teuthology.lock.ops:Start node 'smithi167.front.sepia.ceph.com' reimaging 2023-01-15T13:28:36.398 INFO:teuthology.lock.ops:Updating [smithi167.front.sepia.ceph.com]: reset os type and version on server 2023-01-15T13:28:36.398 INFO:teuthology.lock.ops:Updating smithi167.front.sepia.ceph.com on lock server 2023-01-15T13:28:36.414 INFO:teuthology.lock.ops:Node 'smithi167.front.sepia.ceph.com' reimaging is complete 2023-01-15T13:28:36.558 INFO:teuthology.provision.fog.smithi150:Scheduling deploy of centos 8.stream 2023-01-15T13:28:36.570 INFO:teuthology.provision.fog.smithi167:Scheduling deploy of centos 8.stream 2023-01-15T13:28:36.791 INFO:teuthology.orchestra.console:Power off smithi150 2023-01-15T13:28:36.805 INFO:teuthology.orchestra.console:Power off smithi167 2023-01-15T13:28:45.158 INFO:teuthology.orchestra.console:Power off for smithi150 completed 2023-01-15T13:28:45.164 INFO:teuthology.orchestra.console:Power off for smithi167 completed 2023-01-15T13:28:45.259 INFO:teuthology.orchestra.console:Power on smithi150 2023-01-15T13:28:45.266 INFO:teuthology.orchestra.console:Power on smithi167 2023-01-15T13:28:49.510 INFO:teuthology.orchestra.console:Power on for smithi150 completed 2023-01-15T13:28:49.611 INFO:teuthology.provision.fog.smithi150:Waiting for deploy to finish 2023-01-15T13:28:53.625 INFO:teuthology.orchestra.console:Power on for smithi167 completed 2023-01-15T13:28:53.725 INFO:teuthology.provision.fog.smithi167:Waiting for deploy to finish 2023-01-15T13:31:58.244 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-01-15T13:32:07.304 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-01-15T13:32:09.252 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-01-15T13:32:16.392 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-01-15T13:32:18.664 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-01-15T13:32:25.476 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-01-15T13:32:27.748 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-01-15T13:32:34.532 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-01-15T13:32:36.804 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-01-15T13:32:43.588 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-01-15T13:33:21.482 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-01-15T13:33:42.809 WARNING:teuthology.provision.fog:timed out 2023-01-15T13:33:47.142 INFO:teuthology.provision.fog.smithi167:Node is ready 2023-01-15T13:33:47.166 INFO:teuthology.orchestra.run.smithi167.stdout:smithi167.front.sepia.ceph.com 2023-01-15T13:33:47.227 INFO:teuthology.orchestra.run.smithi167.stdout:172.21.15.167 smithi167.front.sepia.ceph.com smithi167 2023-01-15T13:33:47.610 INFO:teuthology.provision.fog.smithi167:Deploy complete! 2023-01-15T13:33:49.766 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-01-15T13:33:50.038 INFO:teuthology.provision.fog.smithi150:Node is ready 2023-01-15T13:33:50.060 INFO:teuthology.orchestra.run.smithi150.stdout:smithi150.front.sepia.ceph.com 2023-01-15T13:33:50.119 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150 smithi150.front.sepia.ceph.com smithi150 2023-01-15T13:33:50.477 INFO:teuthology.provision.fog.smithi150:Deploy complete! 2023-01-15T13:33:50.677 INFO:teuthology.lock.ops:Checking smithi150.front.sepia.ceph.com 2023-01-15T13:33:50.677 INFO:teuthology.lock.ops:New key found. Updating... 2023-01-15T13:33:50.697 INFO:teuthology.lock.ops:Checking smithi167.front.sepia.ceph.com 2023-01-15T13:33:50.698 INFO:teuthology.lock.ops:New key found. Updating... 2023-01-15T13:33:50.715 INFO:teuthology.lock.ops:Updating [smithi150.front.sepia.ceph.com]: set os type and version on server 2023-01-15T13:33:50.715 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-15T13:33:50.906 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2023-01-15T13:33:50.964 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="CentOS Stream" 2023-01-15T13:33:50.964 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="8" 2023-01-15T13:33:50.964 INFO:teuthology.orchestra.run.smithi150.stdout:ID="centos" 2023-01-15T13:33:50.964 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="rhel fedora" 2023-01-15T13:33:50.964 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="8" 2023-01-15T13:33:50.964 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el8" 2023-01-15T13:33:50.964 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="CentOS Stream 8" 2023-01-15T13:33:50.964 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31" 2023-01-15T13:33:50.964 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-01-15T13:33:50.964 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://centos.org/" 2023-01-15T13:33:50.965 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-01-15T13:33:50.965 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-01-15T13:33:50.965 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-01-15T13:33:50.966 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2023-01-15T13:33:50.981 INFO:teuthology.lock.ops:Updating [smithi167.front.sepia.ceph.com]: set os type and version on server 2023-01-15T13:33:50.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-15T13:33:51.178 INFO:teuthology.orchestra.run.smithi167.stdout:x86_64 2023-01-15T13:33:51.237 INFO:teuthology.orchestra.run.smithi167.stdout:NAME="CentOS Stream" 2023-01-15T13:33:51.237 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION="8" 2023-01-15T13:33:51.237 INFO:teuthology.orchestra.run.smithi167.stdout:ID="centos" 2023-01-15T13:33:51.238 INFO:teuthology.orchestra.run.smithi167.stdout:ID_LIKE="rhel fedora" 2023-01-15T13:33:51.238 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION_ID="8" 2023-01-15T13:33:51.238 INFO:teuthology.orchestra.run.smithi167.stdout:PLATFORM_ID="platform:el8" 2023-01-15T13:33:51.238 INFO:teuthology.orchestra.run.smithi167.stdout:PRETTY_NAME="CentOS Stream 8" 2023-01-15T13:33:51.238 INFO:teuthology.orchestra.run.smithi167.stdout:ANSI_COLOR="0;31" 2023-01-15T13:33:51.238 INFO:teuthology.orchestra.run.smithi167.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-01-15T13:33:51.238 INFO:teuthology.orchestra.run.smithi167.stdout:HOME_URL="https://centos.org/" 2023-01-15T13:33:51.238 INFO:teuthology.orchestra.run.smithi167.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-01-15T13:33:51.238 INFO:teuthology.orchestra.run.smithi167.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-01-15T13:33:51.238 INFO:teuthology.orchestra.run.smithi167.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-01-15T13:33:51.239 INFO:teuthology.lock.ops:Updating smithi167.front.sepia.ceph.com on lock server 2023-01-15T13:33:51.328 INFO:teuthology.dispatcher.supervisor:Running job 7142503 2023-01-15T13:33:51.328 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/bin/teuthology -v --owner scheduled_ivancich@teuthology --archive /home/teuthworker/archive/ivancich-2023-01-15_02:53:50-rgw-wip-eric-testing-2-distro-default-smithi/7142503 --name ivancich-2023-01-15_02:53:50-rgw-wip-eric-testing-2-distro-default-smithi --description rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} -- /home/teuthworker/archive/ivancich-2023-01-15_02:53:50-rgw-wip-eric-testing-2-distro-default-smithi/7142503/orig.config.yaml 2023-01-15T13:33:51.331 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/ivancich-2023-01-15_02:53:50-rgw-wip-eric-testing-2-distro-default-smithi/7142503 2023-01-15T13:33:51.332 INFO:teuthology.dispatcher.supervisor:Job PID: 15014 2023-01-15T13:33:51.332 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-01-16T01:35:12.787 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T01:35:12.934 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T01:35:13.643 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T01:35:13.643 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T01:35:13.643 INFO:teuthology.misc:Compressing logs... 2023-01-16T01:35:13.643 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T01:35:13.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T01:35:14.139 INFO:teuthology.misc:Compressing logs... 2023-01-16T01:35:14.296 INFO:teuthology.misc:Compressing logs... 2023-01-16T01:35:37.295 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/ceph-mon.a.log: file size changed while zipping 2023-01-16T01:36:00.267 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.5.log: file size changed while zipping 2023-01-16T01:36:13.044 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/ceph-osd.2.log: file size changed while zipping 2023-01-16T01:36:46.287 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.7.log: file size changed while zipping 2023-01-16T01:36:56.475 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping 2023-01-16T01:36:58.231 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/ceph-osd.1.log: file size changed while zipping 2023-01-16T01:36:58.670 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/ceph-mgr.y.log: file size changed while zipping 2023-01-16T01:37:00.976 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mgr.x.log: file size changed while zipping 2023-01-16T01:37:37.356 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.6.log: file size changed while zipping 2023-01-16T01:37:50.319 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/ceph-osd.3.log: file size changed while zipping 2023-01-16T01:38:27.421 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.4.log: file size changed while zipping 2023-01-16T01:38:36.849 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/ceph-osd.0.log: file size changed while zipping 2023-01-16T01:38:44.305 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/ceph-mon.c.log: file size changed while zipping 2023-01-16T01:40:29.288 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T01:40:30.077 INFO:teuthology.kill:Nuking machines: ['smithi150', 'smithi167'] 2023-01-16T01:40:30.766 INFO:teuthology.kill:2023-01-16 01:40:30,766.766 INFO:teuthology.nuke:targets: 2023-01-16T01:40:30.766 INFO:teuthology.kill: smithi150.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA9MdZxQwyPtSNGgHCOTpv5VIbXgT7nv0KbytEKW4PhbBus9OLynZREP9imHe9T5aZ8Q0GCtZJdioSX1l78TlR4= 2023-01-16T01:40:30.767 INFO:teuthology.kill: smithi167.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS/y0WOQNzCfQq+Ptb5eFhVUmmtepyuKpeLGzuaW8tRAFEF3zwgKlGJ6UinixUoFtuvdJBzMnSpsZjJJzhY2iKl7Z5Y0uAWeft+l4gqxs3ICPt45PCbG5QrPcqmyg4tsgPwOW68EWhzlnfAdSj08USyvdBvxpOUxgDW/eHKJPkMYK3dk4zeD5+23kT2apnj2jnP8cFoGBWS2ojJC9gSId/WEKj2qpJhfbKAuoyElO/w4mDac3OSkS8L6TLYFtKtOIUWxjjI59o5+m3j8mHFGo/EMNoMdpjXT26DmryDEXy2zCMoEmcZSiRuc3UqI6DpRDzxt7gTLNp8B2CPWOTyr+czWZ55oTSkNMpk7aJCKNpOFVYMlZSb3ie3bFSZamfPkVsKzEhdLVgJQ3Q+SM3lkla9d9Ie5uCGeJfvfBfHOnyruUgJskhIb7Z+1j7qCt76jPJNkdVlBo3gW+UFpROOrw0a1HyojH0Fuh9h86+GQm0ciWNBW+F5km90sQNDIRX8JE= 2023-01-16T01:40:30.767 INFO:teuthology.kill:2023-01-16 01:40:30,766.766 INFO:teuthology.nuke:Checking targets against current locks 2023-01-16T01:40:30.815 INFO:teuthology.kill:2023-01-16 01:40:30,815.815 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-16T01:40:30.840 INFO:teuthology.kill:2023-01-16 01:40:30,840.840 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-16T01:40:30.893 INFO:teuthology.kill:2023-01-16 01:40:30,892.892 INFO:teuthology.orchestra.console:Power off smithi150 2023-01-16T01:40:30.934 INFO:teuthology.kill:2023-01-16 01:40:30,934.934 INFO:teuthology.orchestra.console:Power off smithi167 2023-01-16T01:40:39.276 INFO:teuthology.kill:2023-01-16 01:40:39,275.275 INFO:teuthology.orchestra.console:Power off for smithi150 completed 2023-01-16T01:40:39.302 INFO:teuthology.kill:2023-01-16 01:40:39,302.302 INFO:teuthology.orchestra.console:Power off for smithi167 completed 2023-01-16T01:40:39.420 INFO:teuthology.kill:2023-01-16 01:40:39,420.420 INFO:teuthology.lock.ops:unlocked: smithi150.front.sepia.ceph.com 2023-01-16T01:40:39.441 INFO:teuthology.kill:2023-01-16 01:40:39,441.441 INFO:teuthology.lock.ops:unlocked: smithi167.front.sepia.ceph.com 2023-01-16T01:42:39.684 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T01:42:39.732 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T01:42:40.550 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T01:42:40.550 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T01:42:40.550 INFO:teuthology.misc:Compressing logs... 2023-01-16T01:42:40.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T01:42:40.668 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T01:42:41.743 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T01:42:42.857 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T01:42:43.974 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T01:42:45.050 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T01:42:46.166 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T01:42:47.241 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T01:42:48.355 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T01:42:49.433 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T01:42:50.546 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T01:42:50.546 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi150 2023-01-16T01:42:50.594 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T01:42:51.362 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T01:44:51.412 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T01:44:51.458 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T01:44:52.265 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T01:44:52.266 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T01:44:52.266 INFO:teuthology.misc:Compressing logs... 2023-01-16T01:44:52.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T01:44:55.336 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi150 2023-01-16T01:44:55.383 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T01:44:56.109 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T01:46:56.193 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T01:46:56.240 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T01:46:57.012 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T01:46:57.012 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T01:46:57.012 INFO:teuthology.misc:Compressing logs... 2023-01-16T01:46:57.012 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T01:46:57.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T01:46:57.332 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T01:46:57.503 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T01:46:57.533 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T01:46:57.533 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T01:46:57.573 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T01:46:57.573 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T01:46:57.575 INFO:teuthology.misc:Compressing logs... 2023-01-16T01:46:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T01:46:57.633 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T01:46:57.651 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T01:46:57.651 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T01:46:57.667 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T01:46:57.667 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T01:46:57.669 INFO:teuthology.misc:Compressing logs... 2023-01-16T01:46:57.685 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T01:46:57.685 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T01:46:57.745 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T01:46:57.745 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T01:46:57.763 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T01:46:57.763 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T01:46:57.811 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T01:46:58.568 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T01:48:58.621 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T01:48:58.667 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T01:48:59.409 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T01:48:59.409 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T01:48:59.409 INFO:teuthology.misc:Compressing logs... 2023-01-16T01:48:59.410 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T01:48:59.625 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T01:48:59.637 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T01:48:59.836 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T01:48:59.854 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T01:48:59.854 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T01:48:59.871 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T01:48:59.871 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T01:48:59.872 INFO:teuthology.misc:Compressing logs... 2023-01-16T01:48:59.886 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T01:48:59.888 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T01:48:59.944 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T01:48:59.944 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T01:48:59.962 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T01:48:59.962 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T01:48:59.964 INFO:teuthology.misc:Compressing logs... 2023-01-16T01:48:59.978 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T01:48:59.979 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T01:48:59.992 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T01:48:59.992 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T01:49:00.009 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T01:49:00.010 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T01:49:00.061 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T01:49:08.274 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T01:51:08.325 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T01:51:08.376 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T01:51:09.121 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T01:51:09.121 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T01:51:09.121 INFO:teuthology.misc:Compressing logs... 2023-01-16T01:51:09.121 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T01:51:27.652 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi150 2023-01-16T01:51:27.697 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T01:51:28.419 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T01:53:28.471 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T01:53:28.517 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T01:53:29.279 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T01:53:29.279 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T01:53:29.279 INFO:teuthology.misc:Compressing logs... 2023-01-16T01:53:29.279 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T01:53:29.589 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T01:53:29.906 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T01:53:29.955 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T01:53:29.990 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T01:53:29.990 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T01:53:30.032 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T01:53:30.033 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T01:53:30.038 INFO:teuthology.misc:Compressing logs... 2023-01-16T01:53:30.186 INFO:teuthology.misc:Compressing logs... 2023-01-16T01:53:30.212 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T01:53:30.248 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T01:53:30.278 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T01:53:30.278 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T01:53:30.316 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T01:53:30.316 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T01:53:30.390 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T01:53:31.109 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T01:55:31.161 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T01:55:31.207 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T01:55:31.965 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T01:55:31.965 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T01:55:31.965 INFO:teuthology.misc:Compressing logs... 2023-01-16T01:55:31.966 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T01:55:32.262 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T01:55:32.277 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T01:55:32.593 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T01:55:32.616 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T01:55:32.617 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T01:55:32.658 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T01:55:32.658 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T01:55:32.661 INFO:teuthology.misc:Compressing logs... 2023-01-16T01:55:32.821 INFO:teuthology.misc:Compressing logs... 2023-01-16T01:55:32.838 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T01:55:32.884 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T01:55:32.906 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T01:55:32.906 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T01:55:32.947 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T01:55:32.947 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T01:55:32.994 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T01:55:33.703 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T01:57:33.755 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T01:57:33.802 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T01:57:34.557 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T01:57:34.558 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T01:57:34.558 INFO:teuthology.misc:Compressing logs... 2023-01-16T01:57:34.558 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T01:57:34.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T01:57:34.878 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T01:57:35.169 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T01:57:35.195 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T01:57:35.195 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T01:57:35.234 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T01:57:35.234 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T01:57:35.237 INFO:teuthology.misc:Compressing logs... 2023-01-16T01:57:35.386 INFO:teuthology.misc:Compressing logs... 2023-01-16T01:57:35.410 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T01:57:35.448 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T01:57:35.476 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T01:57:35.476 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T01:57:35.511 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T01:57:35.511 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T01:57:35.555 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T01:57:36.244 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T01:59:36.292 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T01:59:36.334 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T01:59:37.044 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T01:59:37.044 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T01:59:37.044 INFO:teuthology.misc:Compressing logs... 2023-01-16T01:59:37.044 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T01:59:37.381 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T01:59:37.792 INFO:teuthology.misc:Compressing logs... 2023-01-16T01:59:37.946 INFO:teuthology.misc:Compressing logs... 2023-01-16T01:59:38.118 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T01:59:38.796 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:01:38.844 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:01:38.889 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:01:39.692 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:01:39.692 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:01:39.692 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:01:39.692 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:01:40.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T02:01:40.423 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:01:40.570 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:01:40.750 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:01:41.491 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:03:41.544 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:03:41.590 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:03:42.398 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:03:42.398 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:03:42.398 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:03:42.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:03:42.721 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T02:03:43.142 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:03:43.292 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:03:43.497 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/ceph-mon.c.log: file size changed while zipping 2023-01-16T02:03:43.937 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:03:44.625 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:05:44.677 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:05:44.724 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:05:45.448 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:05:45.448 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:05:45.448 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:05:45.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:06:20.068 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi150 2023-01-16T02:06:20.114 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:06:20.860 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:08:20.919 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:08:20.970 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:08:21.719 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:08:21.719 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:08:21.719 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:08:21.719 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:08:21.793 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:08:22.905 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:08:24.017 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:08:25.129 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:08:26.240 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:08:27.350 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:08:28.461 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:08:29.533 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:08:30.641 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:08:31.751 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:08:31.751 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi150 2023-01-16T02:08:31.797 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:08:32.525 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:10:32.577 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:10:32.623 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:10:33.362 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:10:33.363 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:10:33.363 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:10:33.363 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:10:33.548 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T02:10:33.580 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:10:33.758 INFO:teuthology.orchestra.run.smithi167.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:10:33.801 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:10:33.801 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:10:33.837 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:10:33.837 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:10:33.839 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:10:33.871 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2023-01-16T02:10:33.905 INFO:teuthology.orchestra.run.smithi167.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2023-01-16T02:10:33.941 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T02:10:33.941 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:10:33.973 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T02:10:33.973 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:10:33.974 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:10:34.010 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T02:10:34.040 INFO:teuthology.orchestra.run.smithi167.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T02:10:34.078 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:10:34.078 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:10:34.107 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:10:34.107 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:10:34.152 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:10:34.858 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:12:34.917 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:12:34.968 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:12:35.712 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:12:35.712 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:12:35.712 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:12:35.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:12:35.890 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T02:12:35.922 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:12:36.115 INFO:teuthology.orchestra.run.smithi167.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:12:36.148 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:12:36.148 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:12:36.175 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:12:36.175 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:12:36.176 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:12:36.218 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2023-01-16T02:12:36.287 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T02:12:36.287 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:12:36.335 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:12:36.362 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T02:12:36.399 INFO:teuthology.orchestra.run.smithi167.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T02:12:36.430 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:12:36.430 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:12:36.467 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:12:36.467 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:12:36.511 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:12:37.220 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:14:37.271 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:14:37.317 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:14:38.067 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:14:38.067 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:14:38.067 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:14:38.067 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:14:38.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T02:14:38.275 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:14:38.447 INFO:teuthology.orchestra.run.smithi167.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:14:38.481 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:14:38.481 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:14:38.515 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:14:38.516 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:14:38.517 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:14:38.551 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2023-01-16T02:14:38.622 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T02:14:38.622 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:14:38.664 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:14:38.694 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T02:14:38.729 INFO:teuthology.orchestra.run.smithi167.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T02:14:38.766 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:14:38.766 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:14:38.796 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:14:38.797 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:14:38.867 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:14:39.624 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:16:39.673 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:16:39.720 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:16:40.462 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:16:40.463 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:16:40.463 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:16:40.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:16:40.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T02:16:40.662 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:16:40.802 INFO:teuthology.orchestra.run.smithi167.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:16:40.830 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:16:40.830 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:16:40.871 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:16:40.871 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:16:40.873 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:16:40.940 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T02:16:40.942 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T02:16:40.988 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:16:41.774 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:18:41.825 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:18:41.871 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:18:42.697 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:18:42.697 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:18:42.697 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:18:42.697 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:18:42.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T02:18:42.904 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:18:43.086 INFO:teuthology.orchestra.run.smithi167.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:18:43.113 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:18:43.113 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:18:43.154 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:18:43.154 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:18:43.155 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:18:43.224 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T02:18:43.225 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T02:18:43.271 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:18:43.997 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:20:44.051 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:20:44.172 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:20:44.969 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:20:44.969 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:20:44.969 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:20:44.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:20:45.156 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T02:20:45.181 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:20:45.405 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:20:45.405 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:20:45.448 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:20:45.477 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T02:20:45.490 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi150 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T02:20:45.535 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:20:46.300 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:22:46.351 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:22:46.393 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:22:47.134 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:22:47.135 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:22:47.135 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:22:47.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:22:47.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T02:22:47.349 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:22:47.562 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:22:47.562 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:22:47.602 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:22:47.633 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T02:22:47.644 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi150 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T02:22:47.690 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:22:48.362 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:24:48.411 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:24:48.459 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:24:49.165 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:24:49.166 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:24:49.166 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:24:49.166 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:24:49.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T02:24:49.375 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:24:49.593 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:24:49.593 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:24:49.633 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:24:49.663 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T02:24:49.675 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi150 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T02:24:49.721 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:24:50.423 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:26:50.476 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:26:50.528 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:26:51.261 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:26:51.261 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:26:51.261 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:26:51.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:26:51.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T02:26:51.480 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:26:51.702 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:26:51.702 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:26:51.743 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:26:51.775 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T02:26:51.785 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi150 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T02:26:51.833 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:26:52.578 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:28:52.627 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:28:52.676 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:28:53.440 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:28:53.440 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:28:53.441 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:28:53.441 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:28:53.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T02:28:53.598 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:28:53.758 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:28:53.758 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:28:53.801 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:28:53.828 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T02:28:53.843 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi150 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T02:28:53.889 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:28:54.612 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:30:54.659 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:30:54.705 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:30:55.434 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:30:55.434 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:30:55.434 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:30:55.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:30:55.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T02:30:55.862 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:30:55.895 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T02:30:55.904 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi150 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T02:30:55.949 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:30:56.666 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:32:56.719 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:32:56.766 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:32:57.512 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:32:57.513 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:32:57.513 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:32:57.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:33:32.068 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi150 2023-01-16T02:33:32.114 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:33:32.776 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:35:32.829 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:35:32.877 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:35:33.598 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:35:33.598 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:35:33.598 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:35:33.598 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:35:33.717 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:35:34.794 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:35:35.911 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:35:36.990 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:35:38.108 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:35:39.185 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:35:40.303 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:35:41.381 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:35:42.498 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:35:43.575 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:35:43.575 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi150 2023-01-16T02:35:43.622 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:35:44.340 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:37:44.394 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:37:44.469 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:37:45.231 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:37:45.231 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:37:45.231 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:37:45.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:38:03.685 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi150 2023-01-16T02:38:03.730 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:38:04.460 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:40:04.512 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:40:04.559 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:40:05.314 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:40:05.315 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:40:05.315 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:40:05.315 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:40:06.099 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T02:40:06.178 ERROR:teuthology.orchestra.connection:Error authenticating with smithi167.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:40:06.216 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T02:40:07.296 ERROR:teuthology.orchestra.connection:Error authenticating with smithi167.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:40:08.413 ERROR:teuthology.orchestra.connection:Error authenticating with smithi167.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:40:09.492 ERROR:teuthology.orchestra.connection:Error authenticating with smithi167.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:40:10.611 ERROR:teuthology.orchestra.connection:Error authenticating with smithi167.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:40:11.690 ERROR:teuthology.orchestra.connection:Error authenticating with smithi167.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:40:12.808 ERROR:teuthology.orchestra.connection:Error authenticating with smithi167.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:40:13.929 ERROR:teuthology.orchestra.connection:Error authenticating with smithi167.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:40:15.046 ERROR:teuthology.orchestra.connection:Error authenticating with smithi167.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:40:16.163 ERROR:teuthology.orchestra.connection:Error authenticating with smithi167.front.sepia.ceph.com: Authentication failed. 2023-01-16T02:40:16.163 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi167 2023-01-16T02:40:16.213 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:40:17.053 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:42:17.105 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:42:17.152 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:42:17.878 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:42:17.879 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:42:17.879 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:42:17.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:42:18.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T02:42:18.193 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T02:42:50.093 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T02:42:50.123 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:42:50.123 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:42:50.320 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:42:50.320 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:42:50.321 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:42:50.337 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T02:42:50.338 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T02:42:50.398 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T02:42:50.399 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:42:50.416 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T02:42:50.416 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:42:50.419 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:42:50.433 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T02:42:50.434 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T02:42:50.494 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:42:50.494 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:42:50.516 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:42:50.516 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:42:50.571 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:42:51.319 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:44:51.373 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:44:51.419 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:44:52.140 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:44:52.141 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:44:52.141 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:44:52.141 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:44:52.388 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T02:44:52.400 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T02:44:52.847 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T02:44:52.866 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:44:52.866 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:44:52.884 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:44:52.884 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:44:52.886 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:44:52.925 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T02:44:52.925 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T02:44:52.941 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T02:44:52.941 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:44:52.987 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T02:44:52.988 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:44:52.990 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:44:53.001 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T02:44:53.006 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T02:44:53.023 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:44:53.023 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:44:53.039 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:44:53.039 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:44:53.085 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:44:53.779 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:46:53.827 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:46:53.873 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:46:54.627 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:46:54.627 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:46:54.628 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:46:54.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:46:54.876 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T02:46:54.890 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T02:46:56.103 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T02:46:56.129 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:46:56.129 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:46:56.180 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:46:56.180 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:46:56.189 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:46:56.205 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T02:46:56.277 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T02:46:56.304 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T02:46:56.304 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:46:56.372 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T02:46:56.372 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:46:56.381 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:46:56.396 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T02:46:56.470 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T02:46:56.496 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:46:56.496 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:46:56.565 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:46:56.565 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:46:56.619 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:46:57.337 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:48:57.397 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:48:57.494 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:48:58.402 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:48:58.472 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:48:58.472 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:48:58.472 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:48:58.728 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T02:48:58.886 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T02:48:59.334 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T02:48:59.356 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:48:59.495 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:48:59.534 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:48:59.535 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:48:59.542 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:48:59.671 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T02:48:59.728 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T02:48:59.728 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:48:59.799 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:48:59.903 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T02:48:59.937 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T02:48:59.964 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:49:00.042 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:49:00.085 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:49:00.085 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:49:00.139 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:49:00.957 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:51:01.102 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:51:01.166 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:51:01.921 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:51:01.921 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:51:01.921 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:51:01.921 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:51:02.168 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T02:51:02.182 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T02:51:02.627 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T02:51:02.651 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:51:02.651 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:51:02.718 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:51:02.718 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:51:02.727 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:51:02.742 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T02:51:02.845 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T02:51:02.846 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:51:02.915 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:51:02.931 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T02:51:02.963 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T02:51:02.995 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:51:02.995 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:51:03.059 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:51:03.059 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:51:03.132 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:51:03.893 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:53:03.949 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:53:03.996 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:53:04.747 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:53:04.747 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:53:04.748 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:53:04.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:53:05.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T02:53:05.014 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T02:53:05.487 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T02:53:05.510 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:53:05.511 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:53:05.583 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:53:05.583 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:53:05.588 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:53:05.604 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T02:53:05.697 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T02:53:05.697 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:53:05.761 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:53:05.777 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T02:53:05.848 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T02:53:05.870 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:53:05.870 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:53:05.939 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:53:05.939 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:53:05.992 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:53:06.704 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:55:06.772 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:55:06.817 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:55:07.623 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:55:07.623 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:55:07.623 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:55:07.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:55:07.876 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T02:55:07.890 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T02:55:08.350 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T02:55:08.373 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:55:08.373 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:55:08.442 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:55:08.442 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:55:08.449 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:55:08.463 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T02:55:08.557 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T02:55:08.557 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:55:08.646 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:55:08.662 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T02:55:08.691 INFO:teuthology.orchestra.run.smithi167.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T02:55:08.713 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:55:08.713 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:55:08.784 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:55:08.784 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:55:08.838 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:55:09.564 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:57:09.617 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:57:09.666 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:57:10.386 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:57:10.386 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:57:10.387 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:57:10.387 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:57:10.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T02:57:10.666 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T02:57:11.673 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:57:11.673 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:57:11.754 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:57:11.771 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T02:57:11.834 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T02:57:11.842 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:57:11.906 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:57:11.921 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T02:57:12.003 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:57:12.003 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:57:12.131 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:57:12.861 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:59:12.912 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:59:12.980 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:59:13.704 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:59:14.127 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:59:14.127 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:59:14.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T02:59:14.376 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T02:59:14.390 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T02:59:14.878 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:59:14.955 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:59:15.018 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:59:15.034 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T02:59:15.092 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T02:59:15.093 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:59:15.160 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:59:15.175 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T02:59:15.280 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:59:15.281 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:59:15.395 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T02:59:16.114 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:01:16.189 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:01:16.236 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:01:17.033 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:01:17.033 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:01:17.033 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:01:17.033 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:01:17.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:01:17.309 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:01:17.797 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:01:17.798 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:01:17.841 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:01:17.862 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:01:17.950 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:01:17.950 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:01:18.030 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:01:18.047 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:01:18.335 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mon.a.log: file size changed while zipping 2023-01-16T03:01:18.425 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:01:18.440 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:01:18.447 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:01:18.451 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:01:18.460 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:01:18.508 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:01:19.243 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:03:19.293 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:03:19.339 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:03:20.031 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:03:20.031 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:03:20.031 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:03:20.031 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:03:20.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:03:20.309 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:03:20.758 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:03:20.758 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:03:20.809 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:03:20.824 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:03:20.916 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:03:20.917 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:03:20.983 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:03:20.999 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:03:21.042 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:03:21.043 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:03:21.059 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:03:21.068 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:03:21.072 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:03:21.076 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:03:21.083 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:03:21.129 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:03:21.852 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:05:21.900 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:05:21.947 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:05:22.684 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:05:22.684 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:05:22.684 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:05:22.684 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:05:22.931 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:05:22.945 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:05:23.450 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:05:23.450 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:05:23.510 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:05:23.525 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:05:23.587 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:05:23.587 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:05:23.652 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:05:23.668 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:05:23.716 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:05:23.716 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:05:23.729 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:05:23.737 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:05:23.740 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:05:23.743 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:05:23.752 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:05:23.807 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:05:24.539 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:07:24.592 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:07:24.639 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:07:25.365 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:07:25.365 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:07:25.365 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:07:25.365 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:07:25.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:07:25.637 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:07:26.126 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:07:26.126 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:07:26.187 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:07:26.202 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:07:26.306 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:07:26.306 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:07:26.377 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:07:26.392 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:07:26.431 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:07:26.431 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:07:26.432 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:07:26.432 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:07:26.432 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:07:26.432 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:07:26.440 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:07:26.485 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:07:27.173 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:09:27.239 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:09:27.284 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:09:28.028 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:09:28.028 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:09:28.029 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:09:28.029 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:09:28.276 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:09:28.287 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:09:28.774 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:09:28.774 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:09:28.837 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:09:28.854 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:09:28.953 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:09:28.953 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:09:29.021 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:09:29.037 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:09:29.077 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:09:29.077 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:09:29.078 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:09:29.078 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:09:29.078 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:09:29.078 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:09:29.084 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:09:29.129 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:09:29.897 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:11:29.953 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:11:30.001 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:11:30.817 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:11:30.817 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:11:30.817 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:11:30.817 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:11:31.076 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:11:31.090 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:11:31.560 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:11:31.560 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:11:31.619 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:11:31.636 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:11:31.736 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:11:31.736 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:11:31.806 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:11:31.822 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:11:31.902 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:11:31.902 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:11:31.902 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:11:31.902 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:11:31.902 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:11:31.902 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:11:31.910 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:11:31.955 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:11:32.663 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:13:32.715 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:13:32.762 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:13:33.514 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:13:33.514 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:13:33.514 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:13:33.514 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:13:33.769 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:13:33.782 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:13:34.269 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:13:34.269 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:13:34.326 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:13:34.342 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:13:34.440 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:13:34.440 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:13:34.511 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:13:34.527 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:13:34.609 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:13:34.609 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:13:34.609 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:13:34.609 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:13:34.609 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:13:34.609 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:13:34.617 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:13:34.680 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:13:35.403 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:15:35.454 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:15:35.507 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:15:36.273 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:15:36.273 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:15:36.273 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:15:36.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:15:36.527 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:15:36.540 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:15:37.020 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:15:37.020 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:15:37.085 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:15:37.101 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:15:37.158 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:15:37.159 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:15:37.237 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:15:37.253 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:15:37.293 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:15:37.294 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:15:37.294 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:15:37.294 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:15:37.294 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:15:37.294 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:15:37.300 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:15:37.360 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:15:38.065 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:17:38.124 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:17:38.169 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:17:38.931 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:17:38.932 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:17:38.932 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:17:38.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:17:39.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:17:39.240 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:17:39.738 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:17:39.738 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:17:39.802 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:17:39.817 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:17:39.879 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:17:39.880 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:17:39.949 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:17:39.965 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:17:40.011 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:17:40.012 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:17:40.012 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:17:40.012 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:17:40.012 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:17:40.012 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:17:40.016 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:17:40.060 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:17:40.783 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:19:40.834 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:19:40.908 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:19:41.647 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:19:41.648 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:19:41.648 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:19:41.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:19:41.939 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:19:41.953 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:19:42.445 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:19:42.445 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:19:42.509 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:19:42.525 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:19:42.614 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:19:42.614 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:19:42.718 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:19:42.734 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:19:42.814 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:19:42.815 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:19:42.815 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:19:42.815 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:19:42.815 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:19:42.815 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:19:42.821 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:19:42.865 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:19:43.613 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:21:43.689 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:21:43.737 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:21:44.523 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:21:44.523 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:21:44.523 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:21:44.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:21:44.776 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:21:44.788 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:21:45.277 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:21:45.277 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:21:45.360 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:21:45.376 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:21:45.437 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:21:45.437 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:21:45.506 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:21:45.521 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:21:45.561 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:21:45.561 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:21:45.561 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:21:45.561 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:21:45.561 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:21:45.561 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:21:45.568 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:21:45.613 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:21:46.298 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:23:46.348 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:23:46.391 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:23:47.078 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:23:47.078 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:23:47.078 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:23:47.078 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:23:47.324 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:23:47.336 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:23:47.811 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:23:47.811 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:23:47.871 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:23:47.887 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:23:47.948 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:23:47.948 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:23:48.016 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:23:48.031 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:23:48.075 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:23:48.075 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:23:48.075 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:23:48.075 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:23:48.076 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:23:48.076 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:23:48.083 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:23:48.129 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:23:48.906 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:25:48.956 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:25:49.000 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:25:49.727 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:25:49.727 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:25:49.727 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:25:49.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:25:49.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:25:49.992 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:25:50.478 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:25:50.478 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:25:50.540 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:25:50.556 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:25:50.614 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:25:50.614 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:25:50.684 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:25:50.699 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:25:50.737 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:25:50.738 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:25:50.738 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:25:50.738 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:25:50.738 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:25:50.738 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:25:50.745 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:25:50.791 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:25:51.470 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:27:51.534 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:27:51.580 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:27:52.301 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:27:52.301 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:27:52.301 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:27:52.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:27:52.563 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:27:52.577 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:27:53.066 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:27:53.066 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:27:53.123 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:27:53.139 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:27:53.233 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:27:53.233 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:27:53.331 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:27:53.348 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:27:53.391 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:27:53.391 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:27:53.391 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:27:53.391 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:27:53.392 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:27:53.392 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:27:53.400 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:27:53.451 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:27:54.149 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:29:54.197 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:29:54.241 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:29:54.944 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:29:54.944 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:29:54.944 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:29:54.944 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:29:55.199 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:29:55.213 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:29:55.693 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:29:55.693 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:29:55.754 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:29:55.770 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:29:55.829 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:29:55.829 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:29:55.901 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:29:55.918 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:29:55.958 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:29:55.958 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:29:55.958 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:29:55.958 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:29:55.958 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:29:55.958 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:29:55.964 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:29:56.009 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:29:56.686 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:31:56.740 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:31:56.784 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:31:57.509 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:31:57.510 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:31:57.510 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:31:57.510 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:31:57.756 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:31:57.768 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:31:58.259 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:31:58.259 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:31:58.318 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:31:58.334 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:31:58.432 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:31:58.432 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:31:58.503 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:31:58.519 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:31:58.556 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:31:58.557 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:31:58.557 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:31:58.557 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:31:58.557 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:31:58.557 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:31:58.564 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:31:58.606 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:31:59.290 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:33:59.345 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:33:59.390 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:34:00.107 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:34:00.107 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:34:00.108 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:34:00.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:34:00.395 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:34:00.409 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:34:00.889 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:34:00.889 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:34:00.951 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:34:00.966 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:34:01.018 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:34:01.018 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:34:01.084 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:34:01.098 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:34:01.139 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:34:01.139 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:34:01.139 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:34:01.139 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:34:01.139 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:34:01.139 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:34:01.147 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:34:01.192 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:34:01.888 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:36:01.942 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:36:01.989 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:36:02.803 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:36:02.804 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:36:02.804 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:36:02.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:36:03.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:36:03.073 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:36:03.560 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:36:03.560 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:36:03.642 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:36:03.658 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:36:03.754 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:36:03.756 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:36:03.817 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:36:03.833 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:36:03.912 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:36:03.912 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:36:03.913 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:36:03.913 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:36:03.913 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:36:03.913 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:36:03.920 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:36:03.966 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:36:04.692 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:38:04.744 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:38:04.788 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:38:05.550 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:38:05.551 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:38:05.551 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:38:05.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:38:05.808 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:38:05.820 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:38:06.312 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:38:06.312 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:38:06.393 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:38:06.409 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:38:06.461 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:38:06.461 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:38:06.558 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:38:06.574 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:38:06.617 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:38:06.617 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:38:06.617 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:38:06.617 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:38:06.617 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:38:06.617 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:38:06.625 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:38:06.671 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:38:07.399 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:40:07.459 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:40:07.507 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:40:08.241 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:40:08.242 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:40:08.242 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:40:08.242 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:40:08.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:40:08.501 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:40:08.982 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:40:08.982 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:40:09.069 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:40:09.085 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:40:09.146 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:40:09.146 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:40:09.215 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:40:09.230 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:40:09.268 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:40:09.268 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:40:09.268 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:40:09.269 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:40:09.269 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:40:09.269 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:40:09.275 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:40:09.318 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:40:10.013 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:42:10.068 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:42:10.113 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:42:10.815 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:42:10.816 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:42:10.816 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:42:10.816 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:42:11.111 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:42:11.125 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:42:11.607 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:42:11.607 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:42:11.668 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:42:11.684 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:42:11.744 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:42:11.745 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:42:11.813 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:42:11.829 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:42:11.912 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:42:11.912 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:42:11.912 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:42:11.913 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:42:11.913 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:42:11.914 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:42:11.919 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:42:11.963 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:42:12.678 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:44:12.732 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:44:12.779 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:44:13.492 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:44:13.492 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:44:13.493 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:44:13.493 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:44:13.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:44:13.757 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:44:14.228 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:44:14.228 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:44:14.312 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:44:14.328 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:44:14.384 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:44:14.385 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:44:14.454 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:44:14.470 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:44:14.553 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:44:14.553 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:44:14.553 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:44:14.554 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:44:14.554 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:44:14.554 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:44:14.559 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:44:14.603 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:44:15.265 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:46:15.312 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:46:15.362 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:46:16.074 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:46:16.952 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:46:16.952 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:46:16.952 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:46:17.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:46:17.267 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:46:17.726 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:46:17.726 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:46:17.786 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:46:17.802 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:46:17.898 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:46:17.899 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:46:17.967 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:46:17.983 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:46:18.024 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:46:18.024 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:46:18.024 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:46:18.024 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:46:18.024 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:46:18.025 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:46:18.031 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:46:18.073 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:46:18.760 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:48:18.807 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:48:18.852 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:48:19.605 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:48:19.606 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:48:19.606 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:48:19.606 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:48:19.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:48:19.873 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:48:20.360 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:48:20.360 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:48:20.420 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:48:20.437 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:48:20.532 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:48:20.533 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:48:20.600 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:48:20.616 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:48:20.700 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:48:20.700 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:48:20.700 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:48:20.700 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:48:20.701 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:48:20.701 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:48:20.707 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:48:20.755 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:48:21.458 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:50:21.504 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:50:21.550 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:50:22.287 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:50:22.287 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:50:22.287 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:50:22.288 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:50:22.595 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:50:22.609 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:50:23.089 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:50:23.089 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:50:23.152 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:50:23.168 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:50:23.226 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:50:23.226 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:50:23.296 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:50:23.312 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:50:23.354 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:50:23.354 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:50:23.354 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:50:23.355 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:50:23.355 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:50:23.355 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:50:23.362 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:50:23.409 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:50:24.099 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:52:24.148 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:52:24.194 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:52:24.922 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:52:24.922 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:52:24.922 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:52:24.922 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:52:25.176 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:52:25.189 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:52:25.687 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:52:25.687 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:52:25.747 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:52:25.763 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:52:25.812 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:52:25.812 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:52:25.905 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:52:25.921 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:52:25.963 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:52:25.963 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:52:25.963 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:52:25.963 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:52:25.963 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:52:25.963 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:52:25.971 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:52:26.018 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:52:26.764 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:54:26.826 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:54:26.871 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:54:27.598 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:54:27.598 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:54:27.599 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:54:27.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:54:27.852 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:54:27.865 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:54:28.329 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:54:28.329 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:54:28.392 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:54:28.407 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:54:28.460 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:54:28.461 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:54:28.529 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:54:28.545 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:54:28.629 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:54:28.630 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:54:28.630 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:54:28.630 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:54:28.630 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:54:28.630 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:54:28.635 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:54:28.682 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:54:29.398 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:56:29.449 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:56:29.496 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:56:30.252 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:56:30.253 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:56:30.253 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:56:30.253 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:56:30.516 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:56:30.529 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:56:31.011 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:56:31.011 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:56:31.068 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:56:31.086 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:56:31.196 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:56:31.196 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:56:31.266 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:56:31.282 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:56:31.324 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:56:31.324 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:56:31.324 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:56:31.324 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:56:31.324 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:56:31.324 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:56:31.330 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:56:31.376 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:56:32.104 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:58:32.162 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:58:32.208 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:58:32.964 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:58:32.964 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:58:32.964 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:58:32.965 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T03:58:33.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T03:58:33.232 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T03:58:33.706 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:58:33.706 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:58:33.766 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:58:33.782 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T03:58:33.879 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:58:33.879 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:58:33.943 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:58:33.958 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T03:58:34.040 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T03:58:34.040 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T03:58:34.040 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T03:58:34.041 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T03:58:34.041 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T03:58:34.041 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T03:58:34.045 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:58:34.091 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T03:58:34.812 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:00:34.862 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:00:34.908 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:00:35.642 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:00:35.642 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:00:35.642 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:00:35.643 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:00:35.896 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:00:35.907 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:00:36.407 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:00:36.408 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:00:36.466 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:00:36.482 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:00:36.589 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:00:36.590 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:00:36.657 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:00:36.673 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:00:36.748 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:00:36.748 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:00:36.748 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:00:36.748 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:00:36.748 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:00:36.748 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:00:36.755 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:00:36.829 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:00:37.522 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:02:37.582 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:02:37.634 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:02:38.437 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:02:38.438 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:02:38.438 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:02:38.438 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:02:38.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:02:38.700 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:02:39.183 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:02:39.183 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:02:39.241 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:02:39.257 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:02:39.349 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:02:39.349 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:02:39.443 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:02:39.459 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:02:39.500 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:02:39.501 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:02:39.501 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:02:39.501 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:02:39.501 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:02:39.501 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:02:39.507 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:02:39.559 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:02:40.264 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:04:40.313 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:04:40.361 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:04:41.103 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:04:41.103 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:04:41.104 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:04:41.104 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:04:41.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:04:41.373 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:04:41.857 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:04:41.857 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:04:41.919 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:04:41.934 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:04:41.992 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:04:41.992 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:04:42.058 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:04:42.074 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:04:42.151 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:04:42.152 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:04:42.152 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:04:42.152 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:04:42.152 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:04:42.152 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:04:42.158 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:04:42.221 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:04:42.927 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:06:42.978 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:06:43.022 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:06:43.737 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:06:43.737 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:06:43.737 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:06:43.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:06:43.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:06:44.005 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:06:44.491 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:06:44.491 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:06:44.550 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:06:44.565 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:06:44.616 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:06:44.616 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:06:44.703 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:06:44.719 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:06:44.754 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:06:44.755 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:06:44.755 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:06:44.755 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:06:44.755 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:06:44.755 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:06:44.762 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:06:44.808 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:06:45.495 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:08:45.552 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:08:45.598 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:08:46.332 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:08:46.332 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:08:46.333 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:08:46.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:08:46.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:08:46.592 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:08:47.049 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:08:47.050 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:08:47.112 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:08:47.128 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:08:47.184 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:08:47.184 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:08:47.270 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:08:47.286 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:08:47.358 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:08:47.358 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:08:47.358 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:08:47.359 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:08:47.359 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:08:47.359 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:08:47.366 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:08:47.410 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:08:48.119 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:10:48.167 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:10:48.220 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:10:48.955 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:10:48.955 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:10:48.956 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:10:48.956 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:10:49.204 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:10:49.217 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:10:49.695 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:10:49.696 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:10:49.756 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:10:49.771 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:10:49.863 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:10:49.863 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:10:49.931 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:10:49.947 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:10:49.988 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:10:49.989 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:10:49.989 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:10:49.989 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:10:49.989 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:10:49.989 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:10:49.995 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:10:50.040 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:10:50.730 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:12:50.832 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:12:50.878 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:12:51.598 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:12:51.599 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:12:51.599 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:12:51.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:12:51.852 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:12:51.863 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:12:52.334 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:12:52.334 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:12:52.394 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:12:52.410 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:12:52.468 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:12:52.468 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:12:52.539 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:12:52.555 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:12:52.595 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:12:52.595 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:12:52.595 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:12:52.595 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:12:52.596 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:12:52.596 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:12:52.603 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:12:52.648 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:12:53.333 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:14:53.382 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:14:53.429 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:14:54.190 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:14:54.191 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:14:54.191 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:14:54.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:14:54.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:14:54.498 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:14:54.968 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:14:54.968 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:14:55.029 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:14:55.045 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:14:55.142 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:14:55.142 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:14:55.215 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:14:55.230 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:14:55.273 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:14:55.273 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:14:55.273 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:14:55.273 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:14:55.273 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:14:55.273 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:14:55.282 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:14:55.328 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:14:56.068 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:16:56.133 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:16:56.178 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:16:56.909 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:16:56.909 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:16:56.910 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:16:56.910 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:16:57.171 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:16:57.186 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:16:57.677 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:16:57.677 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:16:57.739 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:16:57.755 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:16:57.813 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:16:57.813 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:16:57.886 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:16:57.902 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:16:57.939 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:16:57.939 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:16:57.939 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:16:57.939 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:16:57.940 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:16:57.940 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:16:57.946 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:16:57.992 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:16:58.709 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:18:58.760 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:18:58.816 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:18:59.605 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:18:59.606 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:18:59.606 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:18:59.606 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:18:59.860 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:18:59.872 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:19:00.350 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:19:00.350 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:19:00.439 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:19:00.456 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:19:00.518 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:19:00.518 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:19:00.588 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:19:00.604 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:19:00.648 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:19:00.648 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:19:00.648 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:19:00.648 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:19:00.648 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:19:00.649 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:19:00.655 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:19:00.702 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:19:01.462 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:21:01.517 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:21:01.560 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:21:02.298 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:21:02.298 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:21:02.298 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:21:02.299 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:21:02.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:21:02.554 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:21:03.039 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:21:03.039 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:21:03.099 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:21:03.115 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:21:03.210 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:21:03.210 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:21:03.282 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:21:03.299 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:21:03.372 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:21:03.372 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:21:03.372 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:21:03.372 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:21:03.373 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:21:03.373 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:21:03.381 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:21:03.451 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:21:04.144 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:23:04.193 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:23:04.236 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:23:04.951 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:23:04.951 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:23:04.952 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:23:04.952 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:23:05.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:23:05.267 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:23:05.747 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:23:05.747 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:23:05.809 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:23:05.826 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:23:05.888 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:23:05.888 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:23:05.952 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:23:05.968 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:23:06.010 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:23:06.010 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:23:06.010 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:23:06.010 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:23:06.010 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:23:06.010 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:23:06.018 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:23:06.063 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:23:06.744 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:25:06.795 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:25:06.841 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:25:07.569 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:25:07.569 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:25:07.569 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:25:07.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:25:07.820 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:25:07.833 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:25:08.307 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:25:08.307 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:25:08.370 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:25:08.386 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:25:08.437 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:25:08.437 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:25:08.529 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:25:08.546 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:25:08.591 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:25:08.591 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:25:08.591 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:25:08.591 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:25:08.591 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:25:08.591 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:25:08.598 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:25:08.664 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:25:09.341 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:27:09.396 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:27:09.441 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:27:10.165 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:27:10.165 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:27:10.166 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:27:10.166 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:27:10.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:27:10.430 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:27:10.909 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:27:10.909 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:27:10.998 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:27:11.014 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:27:11.070 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:27:11.070 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:27:11.142 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:27:11.158 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:27:11.201 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:27:11.201 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:27:11.201 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:27:11.201 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:27:11.202 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:27:11.202 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:27:11.208 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:27:11.256 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:27:11.973 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:29:12.036 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:29:12.089 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:29:12.843 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:29:12.843 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:29:12.843 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:29:12.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:29:13.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:29:13.149 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:29:13.631 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:29:13.632 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:29:13.693 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:29:13.709 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:29:13.801 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:29:13.801 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:29:13.878 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:29:13.894 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:29:13.968 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:29:13.968 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:29:13.969 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:29:13.969 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:29:13.969 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:29:13.969 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:29:13.977 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:29:14.027 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:29:14.779 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:31:14.834 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:31:14.879 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:31:15.644 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:31:15.644 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:31:15.644 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:31:15.644 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:31:15.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:31:15.918 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:31:16.384 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:31:16.384 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:31:16.472 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:31:16.489 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:31:16.546 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:31:16.546 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:31:16.616 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:31:16.633 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:31:16.673 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:31:16.674 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:31:16.674 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:31:16.674 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:31:16.674 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:31:16.675 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:31:16.683 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:31:16.726 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:31:17.428 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:33:17.481 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:33:17.529 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:33:18.337 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:33:18.338 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:33:18.338 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:33:18.338 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:33:18.596 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:33:18.609 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:33:19.102 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:33:19.103 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:33:19.165 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:33:19.182 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:33:19.239 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:33:19.240 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:33:19.307 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:33:19.322 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:33:19.401 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:33:19.401 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:33:19.401 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:33:19.401 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:33:19.401 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:33:19.401 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:33:19.408 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:33:19.473 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:33:20.191 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:35:20.241 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:35:20.288 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:35:20.991 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:35:20.991 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:35:20.991 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:35:20.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:35:21.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:35:21.301 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:35:21.783 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:35:21.783 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:35:21.843 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:35:21.859 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:35:21.920 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:35:21.920 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:35:21.984 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:35:21.999 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:35:22.083 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:35:22.083 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:35:22.083 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:35:22.083 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:35:22.083 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:35:22.083 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:35:22.090 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:35:22.137 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:35:22.844 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:37:22.891 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:37:22.938 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:37:23.675 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:37:23.676 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:37:23.676 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:37:23.676 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:37:23.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:37:23.993 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:37:24.467 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:37:24.468 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:37:24.528 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:37:24.543 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:37:24.634 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:37:24.634 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:37:24.722 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:37:24.739 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:37:24.778 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:37:24.778 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:37:24.778 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:37:24.778 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:37:24.778 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:37:24.779 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:37:24.790 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:37:24.840 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:37:25.554 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:39:25.601 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:39:25.647 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:39:26.357 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:39:26.357 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:39:26.357 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:39:26.357 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:39:26.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:39:26.619 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:39:27.114 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:39:27.114 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:39:27.200 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:39:27.216 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:39:27.273 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:39:27.273 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:39:27.342 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:39:27.358 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:39:27.444 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:39:27.444 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:39:27.444 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:39:27.444 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:39:27.445 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:39:27.445 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:39:27.449 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:39:27.503 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:39:28.224 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:41:28.278 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:41:28.323 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:41:29.026 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:41:29.026 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:41:29.026 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:41:29.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:41:29.280 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:41:29.293 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:41:29.762 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:41:29.763 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:41:29.823 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:41:29.838 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:41:29.889 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:41:29.889 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:41:29.954 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:41:29.970 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:41:30.052 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:41:30.052 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:41:30.052 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:41:30.052 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:41:30.052 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:41:30.052 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:41:30.059 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:41:30.103 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:41:30.802 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:43:30.847 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:43:30.917 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:43:31.651 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:43:31.651 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:43:31.651 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:43:31.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:43:31.904 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:43:31.917 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:43:32.394 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:43:32.394 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:43:32.457 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:43:32.474 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:43:32.572 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:43:32.573 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:43:32.640 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:43:32.656 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:43:32.697 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:43:32.697 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:43:32.697 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:43:32.697 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:43:32.697 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:43:32.697 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:43:32.704 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:43:32.749 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:43:33.453 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:45:33.502 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:45:33.545 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:45:34.269 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:45:34.270 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:45:34.270 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:45:34.270 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:45:34.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:45:34.534 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:45:35.012 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:45:35.013 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:45:35.101 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:45:35.117 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:45:35.175 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:45:35.176 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:45:35.262 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:45:35.277 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:45:35.323 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:45:35.323 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:45:35.323 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:45:35.323 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:45:35.323 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:45:35.323 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:45:35.330 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:45:35.372 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:45:36.022 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:47:36.075 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:47:36.117 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:47:36.803 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:47:36.803 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:47:36.804 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:47:36.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:47:37.051 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:47:37.064 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:47:37.555 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:47:37.555 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:47:37.617 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:47:37.631 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:47:37.691 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:47:37.692 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:47:37.778 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:47:37.794 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:47:37.842 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:47:37.842 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:47:37.842 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:47:37.842 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:47:37.842 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:47:37.842 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:47:37.850 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:47:37.896 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:47:38.550 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:49:38.602 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:49:38.646 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:49:39.364 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:49:39.365 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:49:39.365 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:49:39.365 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:49:39.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:49:39.637 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:49:40.122 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:49:40.122 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:49:40.183 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:49:40.199 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:49:40.257 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:49:40.257 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:49:40.326 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:49:40.342 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:49:40.385 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:49:40.385 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:49:40.385 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:49:40.386 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:49:40.386 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:49:40.386 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:49:40.394 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:49:40.439 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:49:41.128 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:51:41.186 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:51:41.238 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:51:41.964 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:51:41.964 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:51:41.964 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:51:41.965 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:51:42.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:51:42.235 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:51:42.703 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:51:42.703 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:51:42.789 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:51:42.805 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:51:42.863 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:51:42.863 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:51:42.934 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:51:42.950 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:51:42.990 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:51:42.990 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:51:42.991 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:51:42.991 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:51:42.991 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:51:42.991 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:51:42.995 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:51:43.040 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:51:43.733 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:53:43.786 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:53:43.831 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:53:44.546 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:53:44.546 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:53:44.546 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:53:44.546 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:53:44.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:53:44.815 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:53:45.303 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:53:45.303 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:53:45.366 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:53:45.382 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:53:45.482 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:53:45.483 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:53:45.557 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:53:45.574 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:53:45.618 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:53:45.618 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:53:45.618 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:53:45.618 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:53:45.618 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:53:45.618 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:53:45.626 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:53:45.676 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:53:46.390 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:55:46.439 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:55:46.486 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:55:47.248 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:55:47.248 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:55:47.248 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:55:47.248 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:55:47.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:55:47.524 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:55:48.001 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:55:48.002 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:55:48.061 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:55:48.077 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:55:48.174 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:55:48.174 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:55:48.245 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:55:48.260 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:55:48.303 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:55:48.303 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:55:48.303 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:55:48.303 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:55:48.303 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:55:48.303 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:55:48.308 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:55:48.353 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:55:49.060 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:57:49.118 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:57:49.166 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:57:49.924 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:57:49.924 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:57:49.924 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:57:49.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:57:50.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:57:50.197 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:57:50.670 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:57:50.670 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:57:50.755 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:57:50.771 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:57:50.870 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:57:50.870 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:57:50.938 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:57:50.955 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:57:51.028 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:57:51.029 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:57:51.029 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:57:51.029 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:57:51.029 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:57:51.029 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:57:51.036 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:57:51.095 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:57:51.805 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:59:51.855 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:59:51.902 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:59:52.662 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:59:52.662 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:59:52.662 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:59:52.662 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T04:59:52.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T04:59:52.926 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T04:59:53.419 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:59:53.419 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:59:53.477 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:59:53.493 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T04:59:53.544 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:59:53.545 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:59:53.614 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:59:53.630 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T04:59:53.710 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T04:59:53.711 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T04:59:53.711 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T04:59:53.711 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T04:59:53.711 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T04:59:53.711 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T04:59:53.719 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:59:53.767 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T04:59:54.496 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:01:54.548 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:01:54.596 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:01:55.378 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:01:55.378 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:01:55.378 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:01:55.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:01:55.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:01:55.644 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:01:56.119 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:01:56.119 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:01:56.207 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:01:56.222 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:01:56.278 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:01:56.278 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:01:56.344 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:01:56.359 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:01:56.446 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:01:56.446 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:01:56.446 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:01:56.446 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:01:56.446 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:01:56.447 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:01:56.454 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:01:56.510 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:01:57.345 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:03:57.425 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:03:57.494 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:03:58.218 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:03:58.218 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:03:58.218 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:03:58.218 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:03:58.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:03:58.490 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:03:58.973 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:03:58.973 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:03:59.060 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:03:59.076 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:03:59.173 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:03:59.173 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:03:59.242 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:03:59.258 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:03:59.334 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:03:59.335 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:03:59.335 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:03:59.335 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:03:59.335 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:03:59.335 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:03:59.343 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:03:59.389 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:04:00.099 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:06:00.147 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:06:00.195 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:06:00.951 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:06:00.951 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:06:00.951 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:06:00.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:06:01.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:06:01.226 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:06:01.715 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:06:01.716 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:06:01.805 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:06:01.820 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:06:01.879 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:06:01.879 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:06:01.968 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:06:01.983 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:06:02.068 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:06:02.069 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:06:02.069 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:06:02.069 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:06:02.069 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:06:02.069 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:06:02.076 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:06:02.122 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:06:02.842 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:08:02.897 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:08:02.950 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:08:03.660 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:08:03.660 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:08:03.660 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:08:03.660 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:08:03.928 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:08:03.942 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:08:04.439 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:08:04.439 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:08:04.498 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:08:04.514 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:08:04.606 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:08:04.607 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:08:04.675 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:08:04.691 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:08:04.731 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:08:04.731 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:08:04.731 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:08:04.731 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:08:04.732 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:08:04.732 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:08:04.739 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:08:04.784 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:08:05.482 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:10:05.534 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:10:05.584 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:10:06.348 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:10:06.348 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:10:06.348 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:10:06.349 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:10:06.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:10:06.605 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:10:07.083 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:10:07.084 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:10:07.166 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:10:07.182 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:10:07.243 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:10:07.244 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:10:07.321 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:10:07.337 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:10:07.386 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:10:07.387 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:10:07.387 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:10:07.387 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:10:07.387 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:10:07.387 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:10:07.394 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:10:07.439 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:10:08.191 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:12:08.244 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:12:08.290 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:12:09.130 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:12:09.131 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:12:09.131 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:12:09.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:12:09.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:12:09.416 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:12:09.896 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:12:09.896 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:12:09.977 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:12:09.993 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:12:10.059 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:12:10.059 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:12:10.127 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:12:10.142 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:12:10.187 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:12:10.188 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:12:10.188 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:12:10.188 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:12:10.188 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:12:10.188 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:12:10.195 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:12:10.245 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:12:10.995 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:14:11.056 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:14:11.111 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:14:11.881 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:14:11.881 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:14:11.881 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:14:11.881 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:14:12.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:14:12.152 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:14:12.653 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:14:12.653 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:14:12.711 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:14:12.727 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:14:12.819 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:14:12.819 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:14:12.886 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:14:12.902 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:14:12.943 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:14:12.943 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:14:12.943 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:14:12.943 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:14:12.943 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:14:12.943 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:14:12.950 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:14:12.996 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:14:13.733 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:16:13.790 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:16:13.836 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:16:14.647 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:16:14.647 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:16:14.648 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:16:14.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:16:14.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:16:14.908 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:16:15.406 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:16:15.406 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:16:15.466 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:16:15.481 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:16:15.584 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:16:15.584 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:16:15.649 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:16:15.664 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:16:15.711 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:16:15.712 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:16:15.712 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:16:15.712 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:16:15.712 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:16:15.712 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:16:15.719 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:16:15.767 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:16:16.505 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:18:16.556 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:18:17.458 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:18:18.221 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:18:18.221 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:18:18.221 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:18:18.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:18:18.478 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:18:18.492 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:18:18.976 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:18:18.977 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:18:19.038 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:18:19.053 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:18:19.111 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:18:19.111 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:18:19.177 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:18:19.194 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:18:19.239 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:18:19.239 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:18:19.239 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:18:19.239 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:18:19.239 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:18:19.239 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:18:19.248 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:18:19.294 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:18:20.038 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:20:22.799 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:20:22.849 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:20:26.094 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:20:26.094 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:20:26.094 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:20:26.094 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:20:26.340 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:20:26.351 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:20:26.826 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:20:26.826 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:20:26.888 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:20:26.903 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:20:26.965 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:20:26.966 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:20:27.031 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:20:27.047 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:20:27.084 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:20:27.084 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:20:27.084 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:20:27.084 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:20:27.084 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:20:27.084 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:20:27.090 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:20:27.134 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:20:27.918 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:22:27.970 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:22:28.015 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:22:28.746 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:22:28.746 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:22:28.746 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:22:28.746 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:22:29.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:22:29.014 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:22:29.493 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:22:29.493 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:22:29.552 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:22:29.568 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:22:29.660 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:22:29.660 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:22:29.727 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:22:29.743 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:22:29.783 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:22:29.783 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:22:29.784 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:22:29.784 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:22:29.784 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:22:29.784 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:22:29.792 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:22:29.834 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:22:30.542 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:24:30.602 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:24:30.650 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:24:31.390 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:24:31.390 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:24:31.390 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:24:31.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:24:31.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:24:31.652 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:24:32.158 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:24:32.159 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:24:32.219 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:24:32.235 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:24:32.284 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:24:32.285 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:24:32.379 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:24:32.394 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:24:32.475 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:24:32.475 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:24:32.476 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:24:32.476 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:24:32.476 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:24:32.476 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:24:32.483 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:24:32.530 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:24:33.263 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:26:33.310 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:26:33.357 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:26:34.085 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:26:34.085 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:26:34.085 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:26:34.085 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:26:34.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:26:34.361 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:26:34.841 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:26:34.841 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:26:34.901 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:26:34.917 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:26:35.022 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:26:35.022 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:26:35.089 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:26:35.104 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:26:35.144 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:26:35.145 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:26:35.145 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:26:35.145 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:26:35.145 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:26:35.146 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:26:35.151 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:26:35.197 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:26:35.934 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:28:35.984 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:28:36.032 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:28:36.762 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:28:36.762 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:28:36.762 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:28:36.763 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:28:37.024 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:28:37.038 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:28:37.511 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:28:37.511 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:28:37.568 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:28:37.584 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:28:37.689 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:28:37.689 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:28:37.758 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:28:37.774 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:28:37.854 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:28:37.854 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:28:37.854 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:28:37.855 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:28:37.855 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:28:37.855 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:28:37.863 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:28:37.915 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:28:38.674 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:30:38.735 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:30:38.780 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:30:39.565 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:30:39.565 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:30:39.565 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:30:39.566 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:30:39.820 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:30:39.832 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:30:40.316 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:30:40.317 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:30:40.396 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:30:40.412 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:30:40.507 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:30:40.507 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:30:40.580 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:30:40.596 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:30:40.637 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:30:40.637 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:30:40.637 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:30:40.637 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:30:40.637 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:30:40.637 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:30:40.647 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:30:40.693 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:30:41.435 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:32:41.486 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:32:41.541 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:32:42.250 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:32:42.251 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:32:42.251 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:32:42.251 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:32:42.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:32:42.516 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:32:42.991 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:32:42.991 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:32:43.054 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:32:43.070 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:32:43.128 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:32:43.129 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:32:43.204 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:32:43.220 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:32:43.257 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:32:43.257 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:32:43.257 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:32:43.257 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:32:43.257 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:32:43.257 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:32:43.262 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:32:43.310 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:32:43.998 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:34:44.052 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:34:44.097 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:34:44.846 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:34:44.846 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:34:44.846 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:34:44.846 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:34:45.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:34:45.109 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:34:45.604 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:34:45.604 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:34:45.665 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:34:45.681 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:34:45.771 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:34:45.772 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:34:45.844 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:34:45.860 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:34:45.901 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:34:45.901 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:34:45.901 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:34:45.901 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:34:45.901 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:34:45.901 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:34:45.908 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:34:45.950 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:34:46.639 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:36:46.689 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:36:46.734 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:36:47.451 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:36:47.451 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:36:47.451 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:36:47.451 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:36:47.716 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:36:47.727 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:36:48.210 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:36:48.211 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:36:48.269 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:36:48.285 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:36:48.378 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:36:48.379 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:36:48.474 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:36:48.490 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:36:48.571 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:36:48.572 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:36:48.572 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:36:48.572 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:36:48.572 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:36:48.572 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:36:48.578 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:36:48.625 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:36:49.369 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:38:49.420 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:38:49.469 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:38:50.237 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:38:50.237 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:38:50.238 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:38:50.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:38:50.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:38:50.508 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:38:50.994 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:38:50.994 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:38:51.052 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:38:51.068 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:38:51.168 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:38:51.168 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:38:51.235 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:38:51.252 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:38:51.296 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:38:51.296 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:38:51.296 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:38:51.296 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:38:51.296 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:38:51.296 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:38:51.304 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:38:51.350 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:38:52.065 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:40:52.114 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:40:52.166 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:40:52.940 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:40:52.940 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:40:52.941 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:40:52.941 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:40:53.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:40:53.205 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:40:53.687 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:40:53.687 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:40:53.748 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:40:53.765 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:40:53.861 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:40:53.861 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:40:53.931 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:40:53.946 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:40:53.989 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:40:53.989 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:40:53.989 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:40:53.989 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:40:53.989 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:40:53.989 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:40:53.996 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:40:54.043 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:40:54.722 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:42:54.777 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:42:54.823 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:42:55.587 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:42:55.587 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:42:55.587 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:42:55.587 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:42:55.835 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:42:55.849 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:42:56.343 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:42:56.343 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:42:56.403 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:42:56.419 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:42:56.524 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:42:56.524 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:42:56.587 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:42:56.602 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:42:56.688 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:42:56.688 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:42:56.688 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:42:56.688 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:42:56.689 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:42:56.689 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:42:56.695 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:42:56.739 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:42:57.438 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:44:57.494 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:44:57.540 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:44:58.294 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:44:58.295 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:44:58.295 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:44:58.295 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:44:58.552 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:44:58.565 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:44:59.045 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:44:59.046 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:44:59.106 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:44:59.122 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:44:59.225 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:44:59.226 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:44:59.289 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:44:59.306 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:44:59.345 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:44:59.345 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:44:59.345 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:44:59.346 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:44:59.346 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:44:59.346 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:44:59.352 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:44:59.422 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:45:00.188 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:47:00.251 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:47:00.306 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:47:01.060 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:47:01.061 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:47:01.061 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:47:01.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:47:01.328 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:47:01.341 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:47:01.845 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:47:01.846 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:47:01.903 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:47:01.919 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:47:02.014 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:47:02.014 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:47:02.103 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:47:02.119 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:47:02.199 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:47:02.199 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:47:02.199 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:47:02.199 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:47:02.199 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:47:02.199 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:47:02.205 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:47:02.257 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:47:03.052 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:49:03.108 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:49:03.153 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:49:03.851 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:49:03.851 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:49:03.851 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:49:03.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:49:04.151 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:49:04.165 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:49:04.658 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:49:04.658 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:49:04.718 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:49:04.735 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:49:04.839 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:49:04.839 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:49:04.908 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:49:04.923 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:49:04.963 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:49:04.963 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:49:04.963 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:49:04.963 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:49:04.963 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:49:04.963 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:49:04.969 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:49:05.015 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:49:05.689 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:51:05.738 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:51:05.784 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:51:06.524 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:51:06.524 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:51:06.524 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:51:06.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:51:06.784 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:51:06.798 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:51:07.297 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:51:07.298 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:51:07.385 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:51:07.401 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:51:07.454 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:51:07.454 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:51:07.524 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:51:07.540 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:51:07.586 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:51:07.586 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:51:07.586 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:51:07.586 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:51:07.586 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:51:07.587 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:51:07.595 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:51:07.640 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:51:08.431 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:53:08.484 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:53:08.530 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:53:09.247 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:53:09.248 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:53:09.248 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:53:09.248 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:53:09.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:53:09.517 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:53:09.997 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:53:09.997 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:53:10.062 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:53:10.078 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:53:10.140 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:53:10.140 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:53:10.204 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:53:10.220 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:53:10.297 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:53:10.298 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:53:10.298 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:53:10.298 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:53:10.298 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:53:10.298 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:53:10.304 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:53:10.349 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:53:11.044 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:55:11.095 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:55:11.142 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:55:11.879 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:55:11.879 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:55:11.879 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:55:11.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:55:12.183 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:55:12.197 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:55:12.689 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:55:12.690 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:55:12.748 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:55:12.764 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:55:12.869 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:55:12.869 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:55:12.946 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:55:12.962 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:55:13.002 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:55:13.003 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:55:13.003 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:55:13.003 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:55:13.003 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:55:13.003 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:55:13.011 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:55:13.060 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:55:13.791 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:57:13.839 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:57:13.885 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:57:14.660 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:57:14.687 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:57:14.687 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:57:14.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:57:14.939 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:57:14.953 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:57:15.446 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:57:15.446 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:57:15.505 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:57:15.521 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:57:15.620 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:57:15.620 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:57:15.688 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:57:15.707 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:57:15.741 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:57:15.741 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:57:15.742 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:57:15.742 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:57:15.742 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:57:15.742 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:57:15.749 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:57:15.795 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:57:16.554 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:59:16.616 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:59:16.688 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:59:17.450 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:59:17.450 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:59:17.450 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:59:17.450 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T05:59:17.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T05:59:17.716 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T05:59:18.189 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:59:18.200 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:59:18.278 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:59:18.294 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T05:59:18.354 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:59:18.354 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:59:18.421 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:59:18.437 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T05:59:18.524 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T05:59:18.524 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T05:59:18.525 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T05:59:18.525 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T05:59:18.525 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T05:59:18.525 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T05:59:18.531 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:59:18.577 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T05:59:19.315 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:01:19.363 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:01:19.409 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:01:20.120 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:01:20.120 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:01:20.120 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:01:20.120 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T06:01:20.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T06:01:20.391 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T06:01:20.901 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:01:20.901 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:01:20.961 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:01:20.975 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T06:01:21.083 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T06:01:21.084 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:01:21.150 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:01:21.166 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T06:01:21.253 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T06:01:21.253 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T06:01:21.253 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T06:01:21.253 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T06:01:21.253 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T06:01:21.253 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T06:01:21.260 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T06:01:21.307 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:01:22.002 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:03:22.053 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:03:22.100 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:03:22.864 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:03:22.864 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:03:22.864 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:03:22.865 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T06:03:23.112 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T06:03:23.124 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T06:03:23.607 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:03:23.607 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:03:23.667 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:03:23.683 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T06:03:23.771 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T06:03:23.771 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:03:23.865 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:03:23.881 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T06:03:23.965 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T06:03:23.965 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T06:03:23.965 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T06:03:23.965 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T06:03:23.965 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T06:03:23.965 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T06:03:23.972 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T06:03:24.031 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:03:24.745 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:05:24.794 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:05:24.841 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:05:25.609 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:05:25.609 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:05:25.609 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:05:25.610 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T06:05:25.856 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T06:05:25.868 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T06:05:26.339 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:05:26.339 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:05:26.426 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:05:26.441 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T06:05:26.528 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T06:05:26.528 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:05:26.621 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:05:26.638 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T06:05:26.672 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T06:05:26.672 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T06:05:26.672 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T06:05:26.672 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T06:05:26.672 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T06:05:26.673 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T06:05:26.680 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T06:05:26.750 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:05:27.453 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:07:27.509 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:07:27.556 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:07:28.375 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:07:28.376 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:07:28.376 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:07:28.376 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T06:07:28.644 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T06:07:28.655 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T06:07:29.153 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:07:29.153 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:07:29.237 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:07:29.253 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T06:07:29.305 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T06:07:29.305 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:07:29.405 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:07:29.421 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T06:07:29.502 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T06:07:29.502 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T06:07:29.502 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T06:07:29.503 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T06:07:29.503 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T06:07:29.503 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T06:07:29.509 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T06:07:29.556 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:07:30.479 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:09:30.533 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:09:30.580 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:09:31.318 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:09:31.318 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:09:31.318 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:09:31.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T06:09:31.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T06:09:31.581 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T06:09:32.077 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:09:32.077 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:09:32.163 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:09:32.179 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T06:09:32.284 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T06:09:32.284 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:09:32.351 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:09:32.367 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T06:09:32.409 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T06:09:32.409 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T06:09:32.409 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T06:09:32.409 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T06:09:32.409 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T06:09:32.409 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T06:09:32.417 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T06:09:32.462 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:09:33.181 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:11:33.228 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:11:33.271 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:11:33.958 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:11:33.958 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:11:33.958 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:11:33.958 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T06:11:34.208 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T06:11:34.220 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T06:11:34.751 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:11:34.751 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:11:34.832 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:11:34.848 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T06:11:34.951 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T06:11:34.951 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:11:35.039 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:11:35.054 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T06:11:35.098 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T06:11:35.098 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T06:11:35.099 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T06:11:35.099 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T06:11:35.099 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T06:11:35.099 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T06:11:35.106 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T06:11:35.150 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:11:35.830 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:13:35.876 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:13:35.921 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:13:36.657 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:13:36.657 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:13:36.657 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:13:36.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T06:13:36.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T06:13:36.934 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T06:13:37.411 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:13:37.411 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:13:37.474 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:13:37.490 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T06:13:37.589 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T06:13:37.589 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:13:37.655 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:13:37.671 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T06:13:37.706 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T06:13:37.706 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T06:13:37.707 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T06:13:37.707 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T06:13:37.707 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T06:13:37.707 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T06:13:37.712 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T06:13:37.754 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:13:38.457 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:15:38.510 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:15:38.553 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:15:39.282 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:15:39.282 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:15:39.283 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:15:39.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T06:15:39.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T06:15:39.549 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T06:15:40.028 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:15:40.028 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:15:40.087 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:15:40.105 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T06:15:40.195 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T06:15:40.195 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:15:40.263 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:15:40.279 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T06:15:40.359 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T06:15:40.360 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T06:15:40.360 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T06:15:40.360 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T06:15:40.360 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T06:15:40.360 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T06:15:40.367 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T06:15:40.413 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:15:41.139 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:17:41.189 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:17:41.232 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:17:41.981 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:17:41.981 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:17:41.981 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:17:41.981 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T06:17:42.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T06:17:42.244 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T06:17:42.717 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:17:42.718 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:17:42.779 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:17:42.795 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T06:17:42.890 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T06:17:42.890 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:17:42.956 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:17:42.973 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T06:17:43.046 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T06:17:43.046 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T06:17:43.046 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T06:17:43.046 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T06:17:43.046 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T06:17:43.046 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T06:17:43.052 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T06:17:43.097 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:17:43.814 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:19:43.865 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:19:43.910 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:19:44.691 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:19:44.692 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:19:44.692 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:19:44.692 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T06:19:44.944 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T06:19:44.957 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T06:19:45.455 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:19:45.456 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:19:45.517 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:19:45.532 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T06:19:45.583 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T06:19:45.584 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:19:45.675 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:19:45.691 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T06:19:45.777 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T06:19:45.777 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T06:19:45.777 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T06:19:45.777 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T06:19:45.777 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T06:19:45.778 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T06:19:45.785 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T06:19:45.831 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:19:46.492 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:21:46.541 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:21:46.589 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:21:47.328 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:21:47.328 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:21:47.328 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:21:47.328 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T06:21:47.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T06:21:47.592 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T06:21:48.064 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:21:48.064 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:21:48.122 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:21:48.138 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T06:21:48.234 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T06:21:48.234 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:21:48.301 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:21:48.317 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T06:21:48.360 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T06:21:48.361 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T06:21:48.361 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T06:21:48.361 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T06:21:48.361 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T06:21:48.361 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T06:21:48.368 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T06:21:48.413 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:21:49.110 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:23:49.158 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:23:49.205 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:23:49.979 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:23:49.979 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:23:49.979 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:23:49.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T06:23:50.240 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T06:23:50.253 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T06:23:50.729 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:23:50.729 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:23:50.813 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:23:50.829 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T06:23:50.930 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T06:23:50.930 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:23:50.995 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:23:51.012 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T06:23:51.098 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T06:23:51.098 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T06:23:51.098 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T06:23:51.098 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T06:23:51.098 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T06:23:51.099 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T06:23:51.106 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T06:23:51.165 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:23:51.847 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:25:51.904 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:25:51.983 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:25:52.777 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:25:52.777 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:25:52.777 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:25:52.777 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T06:25:53.032 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T06:25:53.045 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T06:25:53.560 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:25:53.561 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:25:53.647 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:25:53.663 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T06:25:53.767 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T06:25:53.767 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:25:53.837 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:25:53.853 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T06:25:53.895 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T06:25:53.895 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T06:25:53.895 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T06:25:53.895 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T06:25:53.895 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T06:25:53.895 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T06:25:53.903 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T06:25:53.955 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:25:54.690 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:27:54.747 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:27:54.825 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:27:55.616 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:27:55.617 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:27:55.617 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:27:55.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T06:27:55.884 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T06:27:55.896 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T06:27:56.457 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:27:56.458 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:27:56.519 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:27:56.534 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T06:27:56.639 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T06:27:56.639 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:27:56.709 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:27:56.724 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T06:27:56.802 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T06:27:56.803 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T06:27:56.803 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T06:27:56.803 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T06:27:56.803 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T06:27:56.803 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T06:27:56.810 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T06:27:56.858 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:27:57.607 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:29:57.673 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:29:57.726 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:29:58.532 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:29:58.533 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:29:58.533 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:29:58.533 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T06:29:58.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T06:29:58.800 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T06:29:59.284 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:29:59.284 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:29:59.340 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:29:59.357 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T06:29:59.448 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T06:29:59.449 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:29:59.535 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:29:59.551 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T06:29:59.633 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T06:29:59.633 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T06:29:59.633 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T06:29:59.633 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T06:29:59.633 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T06:29:59.633 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T06:29:59.640 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T06:29:59.686 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:30:00.415 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:32:00.468 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:32:00.518 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:32:01.303 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:32:01.304 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:32:01.304 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:32:01.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T06:32:01.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T06:32:01.575 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T06:32:02.067 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:32:02.067 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:32:02.123 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:32:02.138 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T06:32:02.239 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T06:32:02.239 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:32:02.306 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:32:02.322 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T06:32:02.410 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T06:32:02.410 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T06:32:02.410 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T06:32:02.410 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T06:32:02.410 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T06:32:02.410 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T06:32:02.418 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T06:32:02.466 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:32:03.208 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:34:03.260 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:34:03.309 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:34:04.082 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:34:04.083 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:34:04.083 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:34:04.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-01-16T06:34:04.328 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-01-16T06:34:04.339 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-16T06:34:04.818 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:34:04.819 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:34:04.882 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:34:04.898 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-16T06:34:04.953 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T06:34:04.953 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:34:05.056 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:34:05.071 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/ceph': No such file or directory 2023-01-16T06:34:05.157 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-16T06:34:05.157 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-16T06:34:05.158 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-16T06:34:05.158 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-16T06:34:05.158 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-16T06:34:05.158 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/ceph-osd.2.log.gz already exists; not overwritten 2023-01-16T06:34:05.164 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi167 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T06:34:05.210 INFO:teuthology.kill:Killing Pids: {15014} 2023-01-16T06:34:05.945 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:36:06.201 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...