2023-01-14T00:27:22.221 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2023-01-14T00:27:22.315 INFO:teuthology.lock.ops:Start node 'smithi097.front.sepia.ceph.com' reimaging 2023-01-14T00:27:22.316 INFO:teuthology.lock.ops:Updating [smithi097.front.sepia.ceph.com]: reset os type and version on server 2023-01-14T00:27:22.316 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2023-01-14T00:27:22.333 INFO:teuthology.lock.ops:Node 'smithi097.front.sepia.ceph.com' reimaging is complete 2023-01-14T00:27:22.499 INFO:teuthology.provision.fog.smithi097:Scheduling deploy of centos 8.stream 2023-01-14T00:27:22.746 INFO:teuthology.orchestra.console:Power off smithi097 2023-01-14T00:27:22.895 INFO:teuthology.orchestra.console:Power off for smithi097 completed 2023-01-14T00:27:22.996 INFO:teuthology.orchestra.console:Power on smithi097 2023-01-14T00:27:31.411 INFO:teuthology.orchestra.console:Power on for smithi097 completed 2023-01-14T00:27:31.512 INFO:teuthology.provision.fog.smithi097:Waiting for deploy to finish 2023-01-14T00:30:20.496 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.97 2023-01-14T00:30:29.580 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.97 2023-01-14T00:30:38.668 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.97 2023-01-14T00:30:47.728 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.97 2023-01-14T00:30:56.812 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.97 2023-01-14T00:32:02.876 WARNING:teuthology.provision.fog:timed out 2023-01-14T00:32:09.154 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-01-14T00:32:09.411 INFO:teuthology.provision.fog.smithi097:Node is ready 2023-01-14T00:32:09.469 INFO:teuthology.orchestra.run.smithi097.stdout:smithi097.front.sepia.ceph.com 2023-01-14T00:32:09.529 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97 smithi097.front.sepia.ceph.com smithi097 2023-01-14T00:32:09.880 INFO:teuthology.provision.fog.smithi097:Deploy complete! 2023-01-14T00:32:10.014 INFO:teuthology.lock.ops:Checking smithi097.front.sepia.ceph.com 2023-01-14T00:32:10.015 INFO:teuthology.lock.ops:New key found. Updating... 2023-01-14T00:32:10.036 INFO:teuthology.lock.ops:Updating [smithi097.front.sepia.ceph.com]: set os type and version on server 2023-01-14T00:32:10.037 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T00:32:10.234 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64 2023-01-14T00:32:10.291 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="CentOS Stream" 2023-01-14T00:32:10.292 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="8" 2023-01-14T00:32:10.292 INFO:teuthology.orchestra.run.smithi097.stdout:ID="centos" 2023-01-14T00:32:10.292 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE="rhel fedora" 2023-01-14T00:32:10.293 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="8" 2023-01-14T00:32:10.293 INFO:teuthology.orchestra.run.smithi097.stdout:PLATFORM_ID="platform:el8" 2023-01-14T00:32:10.293 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="CentOS Stream 8" 2023-01-14T00:32:10.293 INFO:teuthology.orchestra.run.smithi097.stdout:ANSI_COLOR="0;31" 2023-01-14T00:32:10.293 INFO:teuthology.orchestra.run.smithi097.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-01-14T00:32:10.293 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://centos.org/" 2023-01-14T00:32:10.294 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-01-14T00:32:10.294 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-01-14T00:32:10.294 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-01-14T00:32:10.295 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2023-01-14T00:32:10.396 INFO:teuthology.dispatcher.supervisor:Running job 7141346 2023-01-14T00:32:10.397 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/bin/teuthology -v --owner scheduled_rishabh@teuthology --archive /home/teuthworker/archive/rishabh-2023-01-13_12:08:33-fs-wip-rishabh-testing-2022Nov24-11Jan2023-distro-default-smithi/7141346 --name rishabh-2023-01-13_12:08:33-fs-wip-rishabh-testing-2022Nov24-11Jan2023-distro-default-smithi --description fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} -- /home/teuthworker/archive/rishabh-2023-01-13_12:08:33-fs-wip-rishabh-testing-2022Nov24-11Jan2023-distro-default-smithi/7141346/orig.config.yaml 2023-01-14T00:32:10.401 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rishabh-2023-01-13_12:08:33-fs-wip-rishabh-testing-2022Nov24-11Jan2023-distro-default-smithi/7141346 2023-01-14T00:32:10.402 INFO:teuthology.dispatcher.supervisor:Job PID: 5905 2023-01-14T00:32:10.403 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-01-14T12:33:40.992 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T12:33:41.170 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:33:41.901 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T12:33:41.901 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:33:41.902 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T12:33:42.271 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:33:42.357 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:33:53.829 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-mon.a.log: file size changed while zipping 2023-01-14T12:34:05.947 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-osd.2.log: file size changed while zipping 2023-01-14T12:34:20.775 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-osd.1.log: file size changed while zipping 2023-01-14T12:34:34.005 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-osd.0.log: file size changed while zipping 2023-01-14T12:34:37.732 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-mgr.x.log: file size changed while zipping 2023-01-14T12:34:49.237 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:34:49.937 INFO:teuthology.kill:Nuking machines: ['smithi097'] 2023-01-14T12:34:50.546 INFO:teuthology.kill:2023-01-14 12:34:50,546.546 INFO:teuthology.nuke:targets: 2023-01-14T12:34:50.547 INFO:teuthology.kill: smithi097.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCl7vAjazyojrx0aBPIvSAluh6GqqP4wet6zhviRzhptM62F3iTcpCCCfUeFyIcIoImUnSe3LgW1rF2KKAPZLGmSQwkBKKSPJ3C6VO/8+nIp6aM/MIzqnkAxaF8JMBsnhNJqoMutmfUknK+ZLoRJgMuJBLKl3g2QTbv2nctlPCB244OX5ZtVA/nss1gQ3/Pq9AhtjbXPfOVGNQsUMsQ6W3bfX+xVYkVp61FbjedkpcLpFZhMy5NcQhUBHxOi2F7rB9MzgrniFts2Gbdts490+kBRTKLUKn6FIdmsXDLvXLDoO+pixtKA0ws2R2VmANQUuc+YYVwVs87OqHcaBmzOH9eeLq+a8OVn2omJqwlmX4Wa9bkG9BP3vcX3V8ngr00/hfNZ1GK0hH9CVwiRpYeEwP8ZGxC59LlqXDlubCeXjQ+pjr4bRzDcxyl7/My3Xg+BsPaUmGP6+NZbJSpjmknwPmOzmEXwtGTKfansrjtSMsNPNPxt1qa8qf5Pj3AmVTarXM= 2023-01-14T12:34:50.547 INFO:teuthology.kill:2023-01-14 12:34:50,546.546 INFO:teuthology.nuke:Checking targets against current locks 2023-01-14T12:34:50.591 INFO:teuthology.kill:2023-01-14 12:34:50,591.591 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-14T12:34:50.655 INFO:teuthology.kill:2023-01-14 12:34:50,655.655 INFO:teuthology.orchestra.console:Power off smithi097 2023-01-14T12:34:59.026 INFO:teuthology.kill:2023-01-14 12:34:59,026.026 INFO:teuthology.orchestra.console:Power off for smithi097 completed 2023-01-14T12:34:59.169 INFO:teuthology.kill:2023-01-14 12:34:59,168.168 INFO:teuthology.lock.ops:unlocked: smithi097.front.sepia.ceph.com 2023-01-14T12:36:59.360 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T12:36:59.412 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:37:00.188 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T12:37:00.189 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:37:00.189 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T12:37:00.269 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T12:37:01.347 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T12:37:02.466 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T12:37:03.581 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T12:37:04.659 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T12:37:05.778 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T12:37:06.854 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T12:37:07.970 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T12:37:09.085 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T12:37:10.201 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T12:37:10.202 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T12:37:10.244 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:37:10.948 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T12:39:11.032 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T12:39:11.079 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:39:11.789 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T12:39:11.789 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:39:11.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T12:39:14.860 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T12:39:14.908 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:39:15.606 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T12:41:15.660 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T12:41:15.708 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:41:16.455 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T12:41:16.456 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:41:16.456 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T12:41:16.663 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-14T12:41:16.740 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T12:41:16.740 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T12:41:16.742 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:41:16.892 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:41:16.957 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-14T12:41:17.025 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T12:41:17.025 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T12:41:17.068 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:41:17.776 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T12:43:17.827 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T12:43:17.873 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:43:18.626 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T12:43:18.626 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:43:18.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T12:43:18.828 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-14T12:43:18.895 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T12:43:18.896 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T12:43:18.897 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:43:19.051 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:43:19.116 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-14T12:43:19.185 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T12:43:19.185 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T12:43:19.228 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:43:19.932 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T12:45:19.995 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T12:45:20.042 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:45:20.781 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T12:45:20.782 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:45:20.782 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T12:45:20.967 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-14T12:45:21.038 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T12:45:21.038 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T12:45:21.039 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:45:21.109 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T12:45:21.111 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T12:45:21.154 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:45:21.868 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T12:47:21.921 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T12:47:21.967 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:47:22.689 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T12:47:22.689 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:47:22.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T12:47:22.895 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-14T12:47:22.963 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T12:47:22.963 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T12:47:22.964 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:47:23.032 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T12:47:23.034 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T12:47:23.111 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:47:23.798 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T12:49:23.868 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T12:49:23.919 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:49:24.663 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T12:49:24.663 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:49:24.664 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T12:49:24.899 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:49:24.965 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T12:49:24.966 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T12:49:25.012 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:49:25.717 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T12:51:25.772 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T12:51:25.823 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:51:26.621 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T12:51:26.622 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:51:26.622 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T12:51:26.907 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:51:26.975 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T12:51:26.977 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T12:51:27.026 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:51:27.702 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T12:53:27.754 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T12:53:27.802 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:53:28.593 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T12:53:28.593 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:53:28.594 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T12:53:28.888 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:53:28.949 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T12:53:28.951 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T12:53:28.998 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:53:29.737 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T12:55:29.792 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T12:55:29.841 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:55:30.630 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T12:55:30.630 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:55:30.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T12:55:30.919 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:55:30.977 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T12:55:30.978 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T12:55:31.024 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:55:31.748 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T12:57:31.801 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T12:57:31.848 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:57:32.590 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T12:57:32.591 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:57:32.592 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T12:57:32.830 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:57:32.898 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T12:57:32.899 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T12:57:32.945 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:57:33.642 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T12:59:33.696 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T12:59:33.742 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:59:34.471 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T12:59:34.472 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:59:34.472 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T12:59:34.715 INFO:teuthology.misc:Compressing logs... 2023-01-14T12:59:34.772 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T12:59:34.774 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T12:59:34.819 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T12:59:35.566 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:01:35.619 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:01:35.664 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:01:36.391 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:01:36.391 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:01:36.392 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:01:36.648 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:01:36.701 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:01:36.703 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:01:36.747 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:01:37.434 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:03:37.506 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:03:37.555 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:03:38.317 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:03:38.317 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:03:38.318 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:03:38.606 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:03:38.665 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:03:38.666 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:03:38.717 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:03:39.451 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:05:39.502 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:05:39.554 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:05:40.264 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:05:40.265 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:05:40.265 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:05:40.551 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:05:40.611 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:05:40.613 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:05:40.656 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:05:41.344 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:07:41.398 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:07:41.447 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:07:42.243 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:07:42.244 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:07:42.244 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:07:42.531 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:07:42.591 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:07:42.593 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:07:42.641 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:07:43.340 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:09:43.400 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:09:43.448 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:09:44.183 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:09:44.184 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:09:44.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:09:44.476 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:09:44.537 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:09:44.539 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:09:44.586 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:09:45.302 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:11:45.354 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:11:45.403 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:11:46.178 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:11:46.179 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:11:46.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:11:46.478 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:11:46.535 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:11:46.537 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:11:46.588 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:11:47.293 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:13:47.343 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:13:47.391 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:13:48.144 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:13:48.145 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:13:48.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:13:48.439 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:13:48.500 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:13:48.502 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:13:48.575 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:13:49.315 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:15:49.365 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:15:49.414 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:15:50.159 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:15:50.159 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:15:50.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:15:50.447 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:15:50.506 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:15:50.507 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:15:50.567 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:15:51.256 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:17:51.316 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:17:51.361 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:17:52.075 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:17:52.075 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:17:52.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:17:52.363 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:17:52.421 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:17:52.423 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:17:52.500 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:17:53.149 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:19:53.198 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:19:53.241 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:19:53.969 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:19:53.970 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:19:53.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:19:54.258 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:19:54.318 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:19:54.319 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:19:54.363 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:19:55.046 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:21:55.100 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:21:55.145 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:21:55.898 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:21:55.899 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:21:55.899 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:21:56.184 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:21:56.242 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:21:56.243 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:21:56.289 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:21:56.993 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:23:57.060 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:23:57.106 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:23:57.872 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:23:57.873 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:23:57.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:23:58.158 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:23:58.215 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:23:58.217 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:23:58.267 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:23:59.003 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:25:59.059 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:25:59.106 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:25:59.924 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:25:59.925 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:25:59.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:26:00.220 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:26:00.272 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:26:00.274 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:26:00.320 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:26:01.031 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:28:01.093 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:28:01.140 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:28:01.876 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:28:01.877 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:28:01.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:28:02.158 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:28:02.215 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:28:02.217 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:28:02.260 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:28:02.926 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:30:02.977 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:30:03.025 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:30:03.769 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:30:03.770 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:30:03.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:30:04.050 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:30:04.107 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:30:04.108 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:30:04.156 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:30:04.929 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:32:04.987 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:32:05.035 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:32:05.795 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:32:05.796 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:32:05.796 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:32:06.086 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:32:06.142 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T13:32:06.143 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:32:06.144 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:32:06.194 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:32:06.891 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:34:06.943 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:34:06.990 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:34:07.715 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:34:07.716 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:34:07.716 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:34:07.956 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:34:08.013 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T13:34:08.013 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:34:08.014 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:34:08.056 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:34:08.767 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:36:08.818 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:36:08.864 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:36:09.589 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:36:09.589 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:36:09.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:36:09.870 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:36:09.926 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T13:36:09.926 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:36:09.927 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:36:09.975 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:36:10.668 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:38:10.723 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:38:10.772 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:38:11.501 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:38:11.502 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:38:11.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:38:11.788 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:38:11.843 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T13:38:11.844 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:38:11.845 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:38:11.894 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:38:12.677 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:40:12.742 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:40:12.789 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:40:13.563 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:40:13.563 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:40:13.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:40:13.856 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:40:13.904 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T13:40:13.904 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:40:13.905 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:40:13.992 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:40:14.722 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:42:14.777 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:42:14.830 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:42:15.643 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:42:15.643 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:42:15.644 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:42:15.928 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:42:15.986 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T13:42:15.986 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:42:15.987 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:42:16.033 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:42:16.760 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:44:16.814 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:44:16.862 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:44:17.604 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:44:17.605 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:44:17.606 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:44:17.889 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:44:17.947 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T13:44:17.947 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:44:17.949 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:44:18.010 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:44:18.725 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:46:18.772 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:46:18.816 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:46:19.538 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:46:19.539 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:46:19.539 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:46:19.824 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:46:19.878 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T13:46:19.878 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:46:19.879 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:46:19.926 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:46:20.605 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:48:20.669 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:48:20.714 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:48:21.451 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:48:21.451 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:48:21.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:48:21.730 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:48:21.788 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T13:48:21.789 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:48:21.790 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:48:21.835 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:48:22.543 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:50:22.598 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:50:22.645 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:50:23.338 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:50:23.339 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:50:23.340 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:50:23.584 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:50:23.641 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T13:50:23.642 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:50:23.643 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:50:23.689 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:50:24.365 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:52:24.421 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:52:24.467 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:52:25.231 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:52:25.231 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:52:25.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:52:25.518 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:52:25.573 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T13:52:25.573 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:52:25.574 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:52:25.622 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:52:26.368 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:54:26.417 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:54:26.464 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:54:27.209 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:54:27.210 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:54:27.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:54:27.497 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:54:27.550 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T13:54:27.550 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:54:27.551 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:54:27.594 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:54:28.325 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:56:28.379 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:56:28.425 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:56:29.204 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:56:29.204 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:56:29.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:56:29.486 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:56:29.543 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T13:56:29.543 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:56:29.544 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:56:29.598 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:56:30.333 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T13:58:30.385 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T13:58:30.433 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:58:31.232 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T13:58:31.233 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:58:31.233 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T13:58:31.521 INFO:teuthology.misc:Compressing logs... 2023-01-14T13:58:31.580 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T13:58:31.580 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T13:58:31.582 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T13:58:31.633 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T13:58:32.355 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:00:32.404 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:00:32.453 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:00:33.278 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:00:33.278 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:00:33.279 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:00:33.562 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:00:33.619 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:00:33.619 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:00:33.620 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:00:33.667 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:00:34.383 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:02:34.434 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:02:34.481 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:02:35.219 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:02:35.220 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:02:35.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:02:35.509 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:02:35.566 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:02:35.567 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:02:35.568 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:02:35.609 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:02:36.336 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:04:36.390 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:04:36.472 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:04:37.203 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:04:37.203 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:04:37.204 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:04:37.483 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:04:37.539 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:04:37.539 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:04:37.540 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:04:37.587 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:04:38.288 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:06:38.344 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:06:38.391 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:06:39.118 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:06:39.119 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:06:39.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:06:39.396 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:06:39.452 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:06:39.452 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:06:39.453 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:06:39.497 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:06:40.189 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:08:40.242 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:08:40.290 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:08:41.032 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:08:41.033 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:08:41.033 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:08:41.316 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:08:41.371 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:08:41.372 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:08:41.373 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:08:41.419 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:08:42.161 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:10:42.213 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:10:42.261 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:10:43.068 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:10:43.069 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:10:43.069 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:10:43.363 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:10:43.413 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:10:43.414 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:10:43.415 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:10:43.462 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:10:44.197 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:12:44.249 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:12:44.304 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:12:45.093 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:12:45.094 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:12:45.094 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:12:45.428 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:12:45.479 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:12:45.479 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:12:45.480 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:12:45.526 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:12:46.218 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:14:46.286 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:14:46.332 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:14:47.050 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:14:47.050 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:14:47.051 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:14:47.334 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:14:47.392 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:14:47.392 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:14:47.393 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:14:47.440 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:14:48.138 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:16:48.191 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:16:48.239 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:16:49.012 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:16:49.013 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:16:49.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:16:49.249 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:16:49.309 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:16:49.309 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:16:49.310 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:16:49.359 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:16:50.149 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:18:50.198 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:18:50.247 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:18:50.944 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:18:50.945 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:18:50.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:18:51.224 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:18:51.283 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:18:51.283 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:18:51.284 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:18:51.330 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:18:52.001 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:20:52.053 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:20:52.099 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:20:52.812 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:20:52.813 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:20:52.813 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:20:53.056 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:20:53.111 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:20:53.111 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:20:53.112 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:20:53.162 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:20:53.878 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:22:53.931 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:22:53.980 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:22:54.726 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:22:54.726 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:22:54.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:22:55.016 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:22:55.066 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:22:55.067 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:22:55.068 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:22:55.114 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:22:55.823 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:24:55.882 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:24:55.928 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:24:56.631 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:24:56.632 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:24:56.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:24:56.916 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:24:56.970 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:24:56.971 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:24:56.972 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:24:57.017 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:24:57.730 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:26:57.779 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:26:57.864 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:26:58.634 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:26:58.635 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:26:58.635 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:26:58.921 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:26:58.980 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:26:58.980 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:26:58.981 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:26:59.028 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:26:59.719 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:28:59.770 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:28:59.818 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:29:00.563 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:29:00.563 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:29:00.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:29:00.848 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:29:00.907 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:29:00.908 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:29:00.909 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:29:00.957 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:29:01.663 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:31:01.720 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:31:01.772 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:31:02.522 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:31:02.523 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:31:02.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:31:02.809 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:31:02.866 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:31:02.867 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:31:02.868 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:31:02.922 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:31:03.640 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:33:03.691 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:33:03.736 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:33:04.456 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:33:04.457 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:33:04.457 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:33:04.742 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:33:04.801 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:33:04.801 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:33:04.802 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:33:04.846 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:33:05.573 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:35:05.626 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:35:05.696 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:35:06.515 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:35:06.515 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:35:06.516 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:35:06.798 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:35:06.856 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:35:06.856 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:35:06.857 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:35:06.903 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:35:07.645 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:37:07.699 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:37:07.744 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:37:08.452 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:37:08.453 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:37:08.453 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:37:08.733 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:37:08.790 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:37:08.790 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:37:08.791 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:37:08.835 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:37:09.554 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:39:09.604 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:39:09.647 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:39:10.393 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:39:10.394 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:39:10.394 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:39:10.678 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:39:10.735 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:39:10.736 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:39:10.737 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:39:10.784 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:39:11.498 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:41:11.550 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:41:11.597 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:41:12.296 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:41:12.297 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:41:12.297 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:41:12.585 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:41:12.641 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:41:12.642 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:41:12.643 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:41:12.686 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:41:13.379 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:43:13.430 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:43:13.477 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:43:14.245 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:43:14.246 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:43:14.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:43:14.532 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:43:14.590 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:43:14.590 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:43:14.591 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:43:14.637 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:43:15.368 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:45:15.432 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:45:15.479 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:45:16.323 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:45:16.324 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:45:16.325 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:45:16.570 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:45:16.623 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:45:16.624 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:45:16.625 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:45:16.672 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:45:17.454 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:47:17.512 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:47:17.562 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:47:18.307 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:47:18.307 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:47:18.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:47:18.596 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:47:18.649 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:47:18.650 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:47:18.652 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:47:18.716 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:47:19.441 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:49:19.499 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:49:19.594 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:49:20.337 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:49:20.337 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:49:20.338 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:49:20.574 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:49:20.633 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:49:20.634 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:49:20.635 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:49:20.681 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:49:21.448 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:51:21.511 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:51:21.559 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:51:22.314 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:51:22.315 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:51:22.315 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:51:22.604 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:51:22.663 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:51:22.663 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:51:22.664 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:51:22.710 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:51:23.384 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:53:23.438 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:53:23.492 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:53:24.335 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:53:24.337 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:53:24.337 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:53:24.623 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:53:24.680 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:53:24.680 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:53:24.681 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:53:24.729 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:53:25.461 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:55:25.512 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:55:25.562 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:55:26.360 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:55:26.361 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:55:26.361 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:55:26.600 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:55:26.658 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:55:26.658 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:55:26.659 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:55:26.704 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:55:27.482 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:57:27.538 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:57:27.586 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:57:28.336 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:57:28.337 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:57:28.338 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:57:28.621 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:57:28.678 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:57:28.679 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:57:28.680 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:57:28.725 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:57:29.513 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T14:59:29.566 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T14:59:29.610 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:59:30.329 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T14:59:30.330 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:59:30.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T14:59:30.614 INFO:teuthology.misc:Compressing logs... 2023-01-14T14:59:30.670 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T14:59:30.670 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T14:59:30.671 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T14:59:30.718 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T14:59:31.433 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:01:31.494 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:01:31.539 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:01:32.231 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:01:32.231 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:01:32.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:01:32.517 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:01:32.573 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:01:32.573 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:01:32.574 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:01:32.620 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:01:33.320 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:03:33.370 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:03:33.418 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:03:34.153 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:03:34.154 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:03:34.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:03:34.437 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:03:34.494 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:03:34.494 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:03:34.495 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:03:34.540 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:03:35.244 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:05:35.301 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:05:35.348 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:05:36.129 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:05:36.130 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:05:36.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:05:36.413 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:05:36.471 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:05:36.471 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:05:36.472 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:05:36.519 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:05:37.261 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:07:37.314 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:07:37.370 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:07:38.138 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:07:38.139 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:07:38.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:07:38.427 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:07:38.483 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:07:38.484 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:07:38.485 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:07:38.534 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:07:39.269 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:09:39.320 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:09:39.367 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:09:40.170 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:09:40.171 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:09:40.171 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:09:40.454 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:09:40.514 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:09:40.514 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:09:40.516 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:09:40.589 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:09:41.390 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:11:41.446 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:11:41.532 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:11:42.275 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:11:42.276 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:11:42.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:11:42.558 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:11:42.615 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:11:42.616 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:11:42.617 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:11:42.665 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:11:43.414 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:13:43.469 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:13:43.517 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:13:44.262 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:13:44.263 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:13:44.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:13:44.552 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:13:44.605 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:13:44.605 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:13:44.607 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:13:44.661 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:13:45.367 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:15:45.418 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:15:45.471 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:15:46.263 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:15:46.264 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:15:46.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:15:46.504 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:15:46.561 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:15:46.561 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:15:46.562 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:15:46.622 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:15:47.380 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:17:47.431 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:17:47.478 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:17:48.176 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:17:48.177 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:17:48.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:17:48.459 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:17:48.516 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:17:48.516 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:17:48.518 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:17:48.571 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:17:49.294 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:19:49.346 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:19:49.389 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:19:50.116 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:19:50.117 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:19:50.117 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:19:50.395 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:19:50.454 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:19:50.454 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:19:50.455 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:19:50.497 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:19:51.144 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:21:51.195 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:21:51.261 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:21:52.062 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:21:52.063 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:21:52.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:21:52.348 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:21:52.403 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:21:52.404 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:21:52.405 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:21:52.450 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:21:53.215 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:23:53.266 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:23:53.315 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:23:54.100 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:23:54.101 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:23:54.101 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:23:54.385 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:23:54.441 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:23:54.442 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:23:54.443 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:23:54.491 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:23:55.200 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:25:55.257 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:25:55.304 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:25:56.085 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:25:56.086 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:25:56.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:25:56.370 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:25:56.428 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:25:56.429 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:25:56.430 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:25:56.482 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:25:57.243 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:27:57.298 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:27:57.357 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:27:58.152 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:27:58.152 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:27:58.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:27:58.439 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:27:58.494 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:27:58.495 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:27:58.496 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:27:58.552 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:27:59.329 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:29:59.385 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:29:59.434 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:30:00.156 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:30:00.157 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:30:00.158 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:30:00.441 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:30:00.498 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:30:00.498 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:30:00.499 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:30:00.545 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:30:01.249 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:32:01.303 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:32:01.345 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:32:02.055 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:32:02.056 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:32:02.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:32:02.336 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:32:02.395 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:32:02.395 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:32:02.396 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:32:02.442 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:32:03.123 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:34:03.174 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:34:03.269 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:34:04.037 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:34:04.038 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:34:04.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:34:04.317 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:34:04.372 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:34:04.372 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:34:04.373 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:34:04.419 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:34:05.157 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:36:05.207 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:36:05.258 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:36:06.060 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:36:06.061 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:36:06.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:36:06.344 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:36:06.399 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:36:06.399 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:36:06.400 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:36:06.449 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:36:07.180 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:38:07.234 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:38:07.281 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:38:08.045 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:38:08.045 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:38:08.046 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:38:08.334 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:38:08.389 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:38:08.389 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:38:08.390 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:38:08.432 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:38:09.133 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:40:09.190 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:40:09.242 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:40:10.006 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:40:10.006 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:40:10.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:40:10.291 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:40:10.349 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:40:10.349 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:40:10.350 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:40:10.397 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:40:11.118 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:42:11.179 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:42:11.227 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:42:12.074 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:42:12.075 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:42:12.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:42:12.399 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:42:12.454 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:42:12.454 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:42:12.455 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:42:12.501 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:42:45.935 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:45:07.457 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:45:07.510 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:45:08.370 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:45:08.371 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:45:08.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:45:08.655 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:45:08.711 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:45:08.711 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:45:08.712 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:45:08.758 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:45:09.519 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:47:10.430 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:47:10.483 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:47:11.245 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:47:11.245 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:47:11.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:47:11.489 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:47:11.546 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:47:11.546 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:47:11.547 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:47:11.595 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:47:12.324 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:49:12.388 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:49:12.432 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:49:13.218 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:49:13.219 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:49:13.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:49:13.501 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:49:13.558 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:49:13.558 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:49:13.559 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:49:13.604 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:49:14.390 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:51:36.302 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:51:36.347 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:51:37.099 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:51:37.099 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:51:37.100 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:51:37.381 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:51:37.436 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:51:37.437 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:51:37.438 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:51:37.483 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:51:38.134 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:53:38.183 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:53:38.230 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:53:38.982 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:53:38.983 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:53:38.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:53:39.275 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:53:39.328 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:53:39.328 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:53:39.329 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:53:39.375 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:53:40.105 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:55:40.157 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:55:40.205 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:55:40.984 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:55:40.985 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:55:40.985 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:55:41.271 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:55:41.330 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:55:41.330 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:55:41.331 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:55:41.378 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:55:42.107 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:57:42.161 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:57:42.245 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:57:42.944 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:57:42.945 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:57:42.945 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:57:43.224 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:57:43.279 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:57:43.280 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:57:43.281 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:57:43.327 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:57:44.067 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T15:59:44.130 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T15:59:44.178 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:59:44.874 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T15:59:44.875 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:59:44.876 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T15:59:45.159 INFO:teuthology.misc:Compressing logs... 2023-01-14T15:59:45.212 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T15:59:45.213 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T15:59:45.214 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T15:59:45.259 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T15:59:45.965 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:01:46.021 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:01:46.069 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:01:46.837 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:01:46.838 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:01:46.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:01:47.129 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:01:47.188 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T16:01:47.188 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T16:01:47.189 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T16:01:47.237 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:01:47.996 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:03:48.050 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:03:48.102 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:03:48.856 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:03:48.857 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:03:48.857 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:03:49.146 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:03:49.204 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T16:03:49.205 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T16:03:49.206 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T16:03:49.252 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:03:49.952 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:05:50.003 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:05:50.050 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:05:50.741 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:05:50.741 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:05:50.742 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:05:51.029 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:05:51.088 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T16:05:51.088 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T16:05:51.090 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T16:05:51.134 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:05:51.835 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:07:51.889 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:07:51.937 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:07:52.680 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:07:52.681 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:07:52.681 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:07:52.968 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:07:53.028 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T16:07:53.028 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T16:07:53.029 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T16:07:53.076 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:07:53.819 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:09:53.870 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:09:53.917 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:09:54.645 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:09:54.646 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:09:54.646 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:09:54.935 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:09:54.994 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T16:09:54.994 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T16:09:54.995 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T16:09:55.042 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:09:55.722 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:11:55.777 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:11:55.824 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:11:56.599 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:11:56.600 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:11:56.601 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:11:56.891 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:11:56.949 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T16:11:56.950 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T16:11:56.951 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T16:11:56.998 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:11:57.688 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:13:57.738 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:13:57.786 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:13:58.625 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:13:58.626 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:13:58.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:13:58.913 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:13:58.975 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T16:13:58.975 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T16:13:58.977 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T16:13:59.024 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:13:59.744 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:15:59.799 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:15:59.846 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:16:00.632 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:16:00.632 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:16:00.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:16:00.958 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:16:01.013 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-14T16:16:01.014 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T16:16:01.015 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T16:16:01.061 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:16:01.786 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:18:01.840 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:18:01.890 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:18:02.665 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:18:02.665 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:18:02.666 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:18:21.008 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T16:18:21.054 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:18:21.775 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:20:21.840 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:20:21.931 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:20:22.670 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:20:22.670 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:20:22.671 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:20:25.420 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T16:20:25.470 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:20:26.189 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:22:26.252 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:22:26.299 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:22:27.096 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:22:27.097 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:22:27.097 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:23:01.584 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T16:23:01.629 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:23:02.390 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:25:02.442 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:25:02.494 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:25:03.297 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:25:03.297 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:25:03.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:26:03.301 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T16:26:03.380 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:26:04.125 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:28:04.185 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:28:04.233 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:28:04.999 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:28:04.999 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:28:05.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:28:05.207 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-14T16:28:05.283 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T16:28:05.283 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T16:28:05.284 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:28:05.429 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:28:05.488 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-14T16:28:05.556 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T16:28:05.556 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T16:28:05.603 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:28:06.331 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:30:06.384 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:30:06.431 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:30:07.200 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:30:07.200 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:30:07.201 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:30:07.407 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-14T16:30:07.475 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T16:30:07.475 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T16:30:07.476 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:30:07.628 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:30:07.680 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-14T16:30:07.747 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T16:30:07.748 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T16:30:07.795 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:30:08.536 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:32:08.594 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:32:08.643 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:32:09.382 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:32:09.382 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:32:09.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:32:09.590 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-14T16:32:09.658 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T16:32:09.659 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T16:32:09.660 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:32:09.809 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:32:09.863 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-14T16:32:09.931 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T16:32:09.932 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T16:32:09.981 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:32:10.682 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:34:10.735 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:34:10.781 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:34:11.576 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:34:11.577 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:34:11.578 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:34:11.786 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-14T16:34:11.854 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T16:34:11.854 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T16:34:11.856 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:34:11.924 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T16:34:11.926 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T16:34:11.972 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:34:12.714 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:36:12.771 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:36:12.818 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:36:13.681 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:36:13.682 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:36:13.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:36:13.986 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:36:14.053 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T16:36:14.055 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T16:36:14.103 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:36:14.844 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:38:14.906 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:38:14.953 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:38:15.809 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:38:15.810 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:38:15.810 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:38:16.137 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:38:16.165 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T16:38:16.166 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T16:38:16.212 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:38:28.501 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:40:28.567 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:40:28.615 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:40:36.399 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:40:36.400 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:40:36.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:40:36.683 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:40:36.748 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T16:40:36.749 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T16:40:36.794 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:40:37.543 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:43:01.816 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:43:01.865 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:43:02.612 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:43:02.613 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:43:02.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:44:02.616 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T16:44:02.663 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:44:03.433 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:46:03.511 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:46:03.557 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:46:04.358 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:46:04.359 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:46:04.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:46:05.900 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T16:46:05.949 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:46:06.775 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:48:06.986 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:48:07.033 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:48:07.778 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:48:07.779 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:48:07.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:48:07.896 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T16:48:09.088 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T16:48:10.202 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T16:48:11.315 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T16:48:12.389 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T16:48:13.502 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T16:48:14.576 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T16:48:15.690 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T16:48:16.763 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T16:48:17.879 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T16:48:17.879 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T16:48:17.925 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:48:18.642 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:50:18.693 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:50:18.743 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:50:19.472 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:50:19.472 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:50:19.473 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:50:19.504 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T16:50:19.587 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:50:20.262 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:52:20.311 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:52:20.358 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:52:21.083 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:52:21.084 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:52:21.084 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:52:21.337 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T16:52:21.394 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T16:52:21.395 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T16:52:21.397 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:52:21.455 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-14T16:52:21.519 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-14T16:52:21.519 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T16:52:21.521 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:52:21.580 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T16:52:21.638 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T16:52:21.638 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T16:52:21.687 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:52:22.447 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:54:22.505 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:54:22.554 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:54:23.297 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:54:23.298 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:54:23.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:54:24.539 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T16:54:24.548 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T16:54:24.548 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T16:54:24.549 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:54:24.598 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-14T16:54:24.660 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-14T16:54:24.660 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T16:54:24.661 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:54:24.719 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T16:54:24.777 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T16:54:24.777 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T16:54:24.823 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:54:25.540 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:56:25.591 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:56:25.654 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:56:26.507 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:56:26.507 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:56:26.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:56:26.747 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T16:56:26.808 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T16:56:26.808 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T16:56:26.810 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:56:26.949 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:56:27.002 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T16:56:27.061 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T16:56:27.061 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T16:56:27.109 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:56:27.839 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T16:58:27.904 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T16:58:27.951 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:58:28.754 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T16:58:28.755 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:58:28.755 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T16:58:29.009 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T16:58:29.070 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T16:58:29.071 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T16:58:29.073 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:58:29.203 INFO:teuthology.misc:Compressing logs... 2023-01-14T16:58:29.246 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T16:58:29.305 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T16:58:29.305 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T16:58:29.355 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T16:58:30.090 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:00:30.137 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:00:30.187 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:00:30.987 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:00:30.988 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:00:30.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:00:31.239 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T17:00:31.298 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T17:00:31.298 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T17:00:31.301 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:00:31.440 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:00:31.490 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T17:00:31.548 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T17:00:31.548 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T17:00:31.596 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:00:32.296 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:02:32.351 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:02:32.409 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:02:33.154 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:02:33.155 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:02:33.156 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:02:51.596 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T17:02:51.651 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:02:52.429 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:04:52.483 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:04:52.529 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:04:53.242 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:04:53.243 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:04:53.243 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:04:56.304 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T17:04:56.349 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:04:57.068 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:06:57.119 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:06:57.175 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:06:57.980 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:06:57.981 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:06:57.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:07:08.108 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T17:07:08.158 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:07:08.929 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:09:08.981 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:09:09.030 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:09:09.866 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:09:09.867 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:09:09.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:09:10.069 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-14T17:09:10.141 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T17:09:10.142 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T17:09:10.143 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:09:10.208 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2023-01-14T17:09:10.275 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-14T17:09:10.276 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T17:09:10.277 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:09:10.343 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-14T17:09:10.410 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T17:09:10.411 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T17:09:10.460 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:09:11.197 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:11:11.258 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:11:11.308 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:11:12.097 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:11:12.097 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:11:12.098 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:11:12.314 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-14T17:11:12.383 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T17:11:12.383 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T17:11:12.384 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:11:12.539 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:11:12.595 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-14T17:11:12.663 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T17:11:12.664 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T17:11:12.716 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:11:13.469 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:13:13.527 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:13:13.576 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:13:14.343 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:13:14.343 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:13:14.344 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:13:14.549 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-14T17:13:14.615 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T17:13:14.615 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T17:13:14.616 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:13:14.772 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:13:14.824 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-14T17:13:14.892 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T17:13:14.893 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T17:13:14.948 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:13:15.705 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:15:15.759 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:15:15.806 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:15:16.533 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:15:16.533 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:15:16.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:15:16.744 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-14T17:15:16.811 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T17:15:16.812 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T17:15:16.813 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:15:16.880 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T17:15:16.881 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T17:15:16.929 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:15:17.649 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:17:17.710 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:17:17.757 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:17:18.560 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:17:18.561 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:17:18.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:17:18.770 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-14T17:17:18.837 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T17:17:18.837 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T17:17:18.839 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:17:18.906 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T17:17:18.908 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T17:17:18.957 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:17:19.664 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:19:19.726 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:19:19.774 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:19:20.535 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:19:20.536 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:19:20.537 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:19:20.818 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:19:20.885 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T17:19:20.887 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T17:19:20.933 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:19:21.652 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:21:21.706 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:21:21.765 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:21:22.579 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:21:22.580 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:21:22.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:21:22.866 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:21:22.935 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T17:21:22.936 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T17:21:22.984 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:21:23.812 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:23:23.864 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:23:23.943 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:23:24.705 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:23:24.705 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:23:24.705 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:23:24.987 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:23:25.056 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T17:23:25.057 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T17:23:25.107 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:23:25.887 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:25:25.942 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:25:25.996 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:25:26.739 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:25:26.739 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:25:26.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:25:27.033 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:25:27.103 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T17:25:27.105 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T17:25:27.159 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:25:27.944 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:27:28.103 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:27:28.152 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:27:28.943 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:27:28.944 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:27:28.945 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:27:29.231 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:27:29.302 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T17:27:29.304 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T17:27:29.360 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:27:30.071 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:29:30.120 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:29:30.170 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:29:30.965 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:29:30.966 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:29:30.966 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:29:31.252 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:29:31.323 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T17:29:31.325 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T17:29:31.373 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:29:32.066 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:31:32.122 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:31:32.180 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:31:32.986 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:31:32.987 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:31:32.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:31:33.275 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:31:33.343 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T17:31:33.344 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T17:31:33.392 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:31:34.190 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:33:34.255 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:33:34.313 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:33:35.075 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:33:35.076 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:33:35.076 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:33:35.352 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:33:35.419 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T17:33:35.420 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T17:33:35.465 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:33:36.155 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:35:36.209 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:35:36.292 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:35:37.031 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:35:37.032 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:35:37.033 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:35:37.325 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:35:37.396 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T17:35:37.397 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T17:35:37.445 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:35:38.155 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:37:38.214 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:37:38.260 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:37:38.999 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:37:39.000 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:37:39.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:37:57.452 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T17:37:57.499 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:37:58.247 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:39:58.298 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:39:58.360 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:39:59.112 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:39:59.112 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:39:59.113 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:40:02.188 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T17:40:02.236 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:40:02.996 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:42:03.046 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:42:03.097 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:42:03.882 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:42:03.883 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:42:03.883 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:42:03.967 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T17:42:05.085 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T17:42:06.203 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T17:42:07.281 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T17:42:08.399 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T17:42:09.477 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T17:42:10.597 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T17:42:11.676 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T17:42:12.793 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T17:42:13.911 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T17:42:13.912 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T17:42:13.959 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:42:14.699 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:44:14.752 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:44:14.800 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:44:15.608 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:44:15.609 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:44:15.609 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:44:16.076 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T17:44:16.128 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:44:16.867 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:46:16.916 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:46:16.965 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:46:17.813 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:46:17.814 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:46:17.814 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:46:17.818 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T17:46:17.864 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:46:18.589 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:48:18.649 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:48:18.712 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:48:19.493 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:48:19.494 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:48:19.494 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:48:19.685 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T17:48:19.711 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T17:48:19.712 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T17:48:19.714 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:48:19.729 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-14T17:48:19.745 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-14T17:48:19.746 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T17:48:19.748 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:48:19.761 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T17:48:19.772 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T17:48:19.772 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T17:48:19.824 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:48:20.537 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:50:20.593 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:50:20.642 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:50:21.519 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:50:21.520 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:50:21.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:50:21.745 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T17:50:21.764 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T17:50:21.764 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T17:50:21.766 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:50:21.815 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:50:21.856 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T17:50:21.868 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T17:50:21.868 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T17:50:21.916 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:50:22.678 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:52:22.730 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:52:22.810 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:52:23.539 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:52:23.539 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:52:23.540 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:52:23.721 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T17:52:23.737 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T17:52:23.737 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T17:52:23.739 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:52:23.828 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:52:23.843 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T17:52:23.906 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T17:52:23.907 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T17:52:23.951 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:52:24.640 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:54:24.687 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:54:24.733 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:54:25.471 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:54:25.472 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:54:25.473 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:54:25.687 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:54:25.732 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:54:25.860 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:54:26.640 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:56:26.695 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:56:26.748 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:56:27.502 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:56:27.503 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:56:27.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:56:27.717 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:56:27.816 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:56:27.953 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-mon.c.log: file size changed while zipping 2023-01-14T17:56:28.103 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:56:28.853 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T17:58:28.904 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T17:58:28.952 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:58:29.706 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T17:58:29.707 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:58:29.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T17:58:29.921 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:58:30.016 INFO:teuthology.misc:Compressing logs... 2023-01-14T17:58:30.038 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T17:58:30.038 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T17:58:30.041 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T17:58:30.087 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T17:58:30.834 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:00:30.886 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:00:30.937 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:00:31.740 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:00:31.740 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:00:31.741 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:00:31.947 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:00:32.007 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:00:32.027 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T18:00:32.028 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T18:00:32.031 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T18:00:32.083 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:00:32.799 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:02:32.856 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:02:32.901 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:02:33.639 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:02:33.640 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:02:33.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:03:08.112 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T18:03:08.163 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:03:08.883 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:05:08.933 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:05:08.989 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:05:09.779 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:05:09.780 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:05:09.781 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:05:09.854 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:05:10.965 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:05:12.038 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:05:13.152 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:05:14.264 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:05:15.376 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:05:16.488 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:05:17.600 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:05:18.712 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:05:19.824 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:05:19.824 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T18:05:19.871 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:05:20.588 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:07:20.651 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:07:20.730 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:07:21.459 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:07:21.460 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:07:21.460 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:07:23.952 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T18:07:24.007 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:07:24.752 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:09:24.805 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:09:24.854 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:09:25.637 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:09:25.638 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:09:25.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:09:25.838 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-14T18:09:25.936 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T18:09:25.937 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T18:09:25.938 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:09:26.054 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:09:26.116 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-14T18:09:26.180 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T18:09:26.181 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T18:09:26.233 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:09:26.984 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:11:27.048 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:11:27.103 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:11:27.879 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:11:27.880 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:11:27.881 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:11:28.090 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-14T18:11:28.153 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T18:11:28.153 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T18:11:28.155 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:11:28.301 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:11:28.349 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-14T18:11:28.414 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T18:11:28.414 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T18:11:28.462 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:11:29.189 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:13:29.243 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:13:29.290 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:13:30.038 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:13:30.039 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:13:30.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:13:30.252 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-14T18:13:30.317 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T18:13:30.317 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T18:13:30.318 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:13:30.383 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T18:13:30.384 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T18:13:30.466 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:13:31.240 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:15:31.292 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:15:31.341 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:15:32.085 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:15:32.085 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:15:32.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:15:32.286 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-14T18:15:32.351 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T18:15:32.351 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T18:15:32.353 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:15:32.417 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T18:15:32.419 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T18:15:32.463 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:15:33.195 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:17:33.250 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:17:33.299 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:17:34.067 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:17:34.068 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:17:34.068 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:17:34.355 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:17:34.423 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T18:17:34.425 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T18:17:34.471 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:17:35.210 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:19:35.260 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:19:35.307 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:19:36.049 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:19:36.049 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:19:36.050 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:19:36.315 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:19:36.379 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T18:19:36.381 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T18:19:36.428 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:19:37.127 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:21:37.182 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:21:37.232 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:21:37.984 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:21:37.984 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:21:37.985 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:21:38.254 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:21:38.317 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T18:21:38.319 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T18:21:38.367 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:21:39.145 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:23:39.204 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:23:39.251 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:23:39.977 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:23:39.977 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:23:39.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:23:40.240 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:23:40.304 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T18:23:40.306 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T18:23:40.352 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:23:41.050 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:25:41.107 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:25:41.158 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:25:41.946 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:25:41.946 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:25:41.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:25:42.226 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:25:42.290 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T18:25:42.292 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T18:25:42.341 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:25:43.070 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:27:43.128 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:27:43.180 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:27:43.958 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:27:43.959 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:27:43.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:27:44.220 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:27:44.284 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T18:27:44.285 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T18:27:44.346 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:27:45.065 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:29:45.120 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:29:45.178 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:29:45.925 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:29:45.926 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:29:45.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:29:46.198 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:29:46.262 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T18:29:46.263 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T18:29:46.310 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:29:47.104 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:31:47.165 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:31:47.222 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:31:47.993 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:31:47.993 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:31:47.994 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:31:48.275 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:31:48.343 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T18:31:48.345 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T18:31:48.393 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:31:49.161 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:33:49.215 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:33:49.265 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:33:50.033 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:33:50.034 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:33:50.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:33:50.297 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:33:50.363 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T18:33:50.365 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T18:33:50.411 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:33:51.121 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:35:51.186 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:35:51.266 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:35:52.126 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:35:52.127 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:35:52.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:35:52.402 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:35:52.469 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T18:35:52.470 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T18:35:52.553 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:35:53.317 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:37:53.372 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:37:53.421 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:37:54.177 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:37:54.178 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:37:54.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:38:12.688 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T18:38:12.735 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:38:13.490 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:40:13.538 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:40:13.586 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:40:14.419 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:40:14.420 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:40:14.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:40:14.494 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:40:15.608 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:40:16.725 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:40:17.797 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:40:18.909 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:40:20.020 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:40:21.095 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:40:22.167 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:40:23.281 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:40:24.394 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:40:24.395 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T18:40:24.443 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:40:25.170 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:42:25.219 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:42:25.267 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:42:26.002 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:42:26.003 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:42:26.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:42:57.969 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T18:42:57.979 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T18:42:57.979 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T18:42:57.980 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:42:58.033 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-14T18:42:58.093 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-14T18:42:58.093 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T18:42:58.095 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:42:58.151 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T18:42:58.209 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T18:42:58.209 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T18:42:58.267 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:42:59.020 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:44:59.072 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:44:59.119 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:44:59.896 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:44:59.896 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:44:59.897 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:45:16.364 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T18:45:16.409 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:45:17.144 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:47:17.198 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:47:17.247 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:47:18.016 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:47:18.017 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:47:18.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:47:18.258 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T18:47:18.313 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T18:47:18.313 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T18:47:18.314 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:47:18.453 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:47:18.493 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T18:47:18.547 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T18:47:18.547 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T18:47:18.617 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:47:19.360 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:49:19.414 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:49:19.461 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:49:20.178 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:49:20.178 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:49:20.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:49:20.435 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T18:49:20.499 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T18:49:20.500 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T18:49:20.501 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:49:20.644 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:49:20.676 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T18:49:20.735 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T18:49:20.735 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T18:49:20.785 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:49:21.529 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:51:21.583 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:51:21.631 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:51:22.401 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:51:22.402 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:51:22.402 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:51:22.640 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T18:51:22.695 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T18:51:22.696 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T18:51:22.698 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:51:22.829 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:51:22.873 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T18:51:22.932 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T18:51:22.933 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T18:51:23.001 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:51:23.695 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:53:23.754 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:53:23.801 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:53:24.654 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:53:24.655 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:53:24.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:53:24.909 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T18:53:24.968 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T18:53:24.969 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T18:53:24.971 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:53:25.111 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:53:25.152 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T18:53:25.212 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T18:53:25.213 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T18:53:25.262 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:53:25.994 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:55:26.057 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:55:26.121 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:55:26.952 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:55:26.953 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:55:26.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:55:45.356 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T18:55:45.405 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:55:46.172 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:57:46.223 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:57:46.273 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:57:47.116 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:57:47.117 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:57:47.117 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T18:57:47.201 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:57:48.280 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:57:49.401 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:57:50.519 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:57:51.599 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:57:52.719 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:57:53.798 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:57:54.915 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:57:55.996 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:57:57.117 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T18:57:57.118 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T18:57:57.165 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:57:57.917 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T18:59:57.971 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T18:59:58.050 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T18:59:58.791 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T18:59:58.792 INFO:teuthology.misc:Compressing logs... 2023-01-14T18:59:58.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:00:30.284 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T19:00:30.332 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:00:31.043 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:02:31.090 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:02:31.137 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:02:31.926 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:02:31.926 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:02:31.927 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:03:31.930 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T19:03:31.979 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:03:32.691 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:05:32.744 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:05:32.795 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:05:33.574 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:05:33.575 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:05:33.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:05:33.833 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T19:05:33.887 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T19:05:33.888 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T19:05:33.889 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:05:38.343 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:05:38.391 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T19:05:38.451 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T19:05:38.451 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T19:05:38.501 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:05:39.355 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:07:39.408 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:07:39.468 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:07:40.249 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:07:40.250 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:07:40.250 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:07:40.499 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T19:07:40.557 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T19:07:40.558 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T19:07:40.559 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:07:40.689 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:07:40.724 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T19:07:40.787 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T19:07:40.787 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T19:07:40.835 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:07:41.595 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:09:41.653 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:09:41.703 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:09:42.488 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:09:42.489 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:09:42.489 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:09:42.743 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T19:09:42.803 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T19:09:42.803 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T19:09:42.805 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:09:42.938 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:09:42.987 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T19:09:43.045 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T19:09:43.045 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T19:09:43.097 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:09:43.821 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:11:43.877 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:11:43.929 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:11:44.720 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:11:44.721 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:11:44.721 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:12:44.725 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T19:12:44.772 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:12:45.504 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:14:45.559 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:14:45.609 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:14:46.389 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:14:46.390 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:14:46.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:15:01.837 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T19:15:02.952 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T19:15:04.068 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T19:15:05.196 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T19:15:06.319 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T19:15:07.436 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T19:15:08.549 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T19:15:09.663 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T19:15:10.737 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T19:15:11.854 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T19:15:11.855 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T19:15:11.904 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:15:12.672 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:17:12.728 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:17:12.776 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:17:13.614 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:17:13.615 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:17:13.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:17:16.688 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T19:17:16.734 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:17:17.469 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:19:17.525 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:19:17.574 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:19:18.329 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:19:18.330 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:19:18.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:19:18.532 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-14T19:19:18.620 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T19:19:18.621 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T19:19:18.622 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:19:18.781 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:19:18.826 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-14T19:19:18.894 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T19:19:18.894 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T19:19:18.944 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:19:19.709 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:21:19.762 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:21:19.812 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:21:20.555 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:21:20.555 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:21:20.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:21:20.722 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-14T19:21:20.788 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T19:21:20.789 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T19:21:20.790 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:21:20.945 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:21:20.993 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-14T19:21:21.061 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T19:21:21.062 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T19:21:21.113 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:21:21.866 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:23:21.923 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:23:21.971 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:23:22.738 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:23:22.739 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:23:22.739 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:23:22.955 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-14T19:23:23.023 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T19:23:23.023 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T19:23:23.025 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:23:23.093 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T19:23:23.094 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T19:23:23.178 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:23:23.951 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:25:24.013 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:25:24.070 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:25:24.858 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:25:24.858 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:25:24.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:25:25.079 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-14T19:25:25.150 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T19:25:25.150 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T19:25:25.152 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:25:25.221 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T19:25:25.222 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T19:25:25.269 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:25:26.031 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:27:26.086 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:27:26.137 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:27:26.915 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:27:26.916 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:27:26.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:27:27.122 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-14T19:27:27.190 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T19:27:27.190 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T19:27:27.192 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:27:27.258 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-14T19:27:27.260 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T19:27:27.303 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:27:28.043 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:29:28.098 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:29:28.151 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:29:28.987 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:29:28.987 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:29:28.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:30:28.992 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T19:30:29.039 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:30:29.752 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:32:29.812 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:32:29.861 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:32:30.578 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:32:30.579 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:32:30.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:32:30.696 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T19:32:31.809 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T19:32:32.922 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T19:32:34.036 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T19:32:35.109 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T19:32:36.224 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T19:32:37.299 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T19:32:38.415 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T19:32:39.489 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T19:32:40.602 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T19:32:40.603 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T19:32:40.654 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:32:41.464 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:34:41.516 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:34:41.566 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:34:42.255 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:34:42.256 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:34:42.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:34:44.176 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T19:34:44.224 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:34:45.006 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:36:45.065 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:36:45.113 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:36:45.843 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:36:45.844 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:36:45.844 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:36:46.024 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T19:36:46.056 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T19:36:46.056 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T19:36:46.058 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:36:46.071 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-14T19:36:46.083 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-14T19:36:46.083 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T19:36:46.084 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:36:46.095 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T19:36:46.155 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T19:36:46.155 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T19:36:46.198 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:36:46.892 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:38:46.946 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:38:46.994 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:38:47.772 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:38:47.773 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:38:47.773 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:39:20.872 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T19:39:20.929 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T19:39:20.929 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T19:39:20.932 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:39:20.946 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-14T19:39:20.959 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-14T19:39:20.960 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T19:39:20.963 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:39:21.022 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T19:39:21.038 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T19:39:21.038 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T19:39:21.106 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:39:21.852 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:41:21.911 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:41:21.957 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:41:22.713 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:41:22.714 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:41:22.714 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:41:22.898 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T19:41:22.914 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T19:41:22.914 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T19:41:22.916 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:41:22.969 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:41:23.011 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T19:41:23.029 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T19:41:23.029 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T19:41:23.076 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:41:23.798 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:43:23.855 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:43:23.908 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:43:24.724 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:43:24.724 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:43:24.725 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:43:24.910 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T19:43:24.926 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T19:43:24.926 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T19:43:24.928 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:43:24.975 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:43:24.988 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T19:43:25.048 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T19:43:25.048 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T19:43:25.095 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:43:25.882 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:45:25.939 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:45:25.982 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:45:26.663 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:45:26.664 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:45:26.664 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:45:26.862 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:45:26.952 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:45:27.023 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:45:27.678 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:47:27.738 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:47:27.787 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:47:28.596 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:47:28.596 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:47:28.596 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:47:28.800 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:47:28.857 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:47:29.014 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:47:29.753 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:49:29.805 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:49:29.852 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:49:30.630 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:49:30.630 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:49:30.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:49:30.829 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:49:30.869 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:49:31.009 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping 2023-01-14T19:49:31.010 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2023-01-14T19:49:31.115 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-osd.6.log.gz already exists; not overwritten 2023-01-14T19:49:33.625 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-mds.b.log: file size changed while zipping 2023-01-14T19:49:33.626 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2023-01-14T19:49:33.892 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-osd.7.log: file size changed while zipping 2023-01-14T19:49:33.894 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T19:49:33.944 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:49:34.719 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:51:34.776 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:51:34.825 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:51:35.579 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:51:35.580 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:51:35.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:51:35.782 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:51:35.879 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:51:35.897 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2023-01-14T19:51:35.897 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2023-01-14T19:51:35.898 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T19:51:35.898 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-osd.6.log.gz already exists; not overwritten 2023-01-14T19:51:35.898 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2023-01-14T19:51:35.898 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T19:51:35.898 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2023-01-14T19:51:35.899 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T19:51:35.944 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:51:36.711 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:53:36.764 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:53:36.813 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:53:37.578 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:53:37.578 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:53:37.578 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:53:37.794 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:53:37.891 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:53:37.912 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2023-01-14T19:53:37.913 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2023-01-14T19:53:37.913 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T19:53:37.914 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-osd.6.log.gz already exists; not overwritten 2023-01-14T19:53:37.914 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2023-01-14T19:53:37.914 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T19:53:37.914 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2023-01-14T19:53:37.915 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T19:53:37.975 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:53:38.698 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:55:38.749 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:55:38.798 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:55:39.577 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:55:39.578 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:55:39.578 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:55:39.780 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:55:39.874 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:55:39.940 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2023-01-14T19:55:39.941 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2023-01-14T19:55:39.941 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T19:55:39.941 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-osd.6.log.gz already exists; not overwritten 2023-01-14T19:55:39.942 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2023-01-14T19:55:39.942 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T19:55:39.942 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2023-01-14T19:55:39.943 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T19:55:39.990 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:55:40.743 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:57:40.796 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:57:40.853 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:57:41.638 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:57:41.638 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:57:41.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T19:57:41.845 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:57:41.954 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:57:41.975 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2023-01-14T19:57:41.976 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists; not overwritten 2023-01-14T19:57:41.976 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T19:57:41.976 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-osd.6.log.gz already exists; not overwritten 2023-01-14T19:57:41.977 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2023-01-14T19:57:41.977 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T19:57:41.977 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists; not overwritten 2023-01-14T19:57:41.978 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T19:57:42.025 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:57:42.759 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T19:59:42.814 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T19:59:42.861 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T19:59:43.647 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T19:59:43.647 INFO:teuthology.misc:Compressing logs... 2023-01-14T19:59:43.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:00:02.000 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T20:00:02.048 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:00:02.812 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:02:02.869 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:02:02.919 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:02:03.722 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:02:03.723 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:02:03.723 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:02:06.800 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T20:02:06.857 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:02:07.600 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:04:07.666 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:04:07.713 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:04:08.492 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:04:08.492 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:04:08.493 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:04:08.573 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T20:04:09.690 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T20:04:10.769 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T20:04:11.886 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T20:04:13.002 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T20:04:14.078 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T20:04:15.193 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T20:04:16.309 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T20:04:17.425 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T20:04:18.542 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed. 2023-01-14T20:04:18.542 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T20:04:18.588 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:04:19.362 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:06:19.431 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:06:19.478 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:06:20.262 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:06:20.263 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:06:20.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:06:20.512 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T20:06:20.550 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T20:06:20.550 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T20:06:20.552 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:06:20.566 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-14T20:06:20.583 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-14T20:06:20.583 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T20:06:20.585 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:06:20.641 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T20:06:20.652 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T20:06:20.652 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T20:06:20.701 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:06:21.411 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:08:21.484 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:08:21.534 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:08:22.368 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:08:22.369 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:08:22.369 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:09:22.372 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi097 2023-01-14T20:09:22.425 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:09:23.180 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:11:23.234 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:11:23.319 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:11:24.065 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:11:24.065 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:11:24.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:11:24.261 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T20:11:24.292 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T20:11:24.292 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T20:11:24.294 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:11:24.356 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:11:24.373 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T20:11:24.436 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T20:11:24.436 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T20:11:24.487 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:11:25.185 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:13:25.239 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:13:25.287 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:13:26.132 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:13:26.133 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:13:26.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:13:26.316 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T20:13:26.328 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T20:13:26.328 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T20:13:26.330 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:13:26.430 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:13:26.446 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T20:13:26.461 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T20:13:26.462 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T20:13:26.510 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:13:27.283 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:15:27.348 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:15:27.399 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:15:28.182 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:15:28.183 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:15:28.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:15:28.409 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T20:15:28.425 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T20:15:28.426 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T20:15:28.427 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:15:28.472 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:15:28.489 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T20:15:28.548 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T20:15:28.548 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-01-14T20:15:28.602 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:15:29.369 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:17:29.426 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:17:29.473 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:17:30.203 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:17:30.204 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:17:30.204 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:17:30.411 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:17:30.449 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:17:30.573 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:17:31.292 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:19:31.346 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:19:31.424 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:19:32.219 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:19:32.220 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:19:32.221 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:19:32.430 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:19:32.523 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:19:32.693 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-osd.5.log: file size changed while zipping 2023-01-14T20:19:32.835 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-osd.6.log: file size changed while zipping 2023-01-14T20:19:32.950 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-osd.4.log: file size changed while zipping 2023-01-14T20:19:33.046 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph-osd.7.log: file size changed while zipping 2023-01-14T20:19:33.242 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:19:34.021 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:21:34.075 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:21:34.124 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:21:34.968 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:21:34.969 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:21:34.969 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:21:35.169 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:21:35.262 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:21:35.281 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T20:21:35.281 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T20:21:35.283 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T20:21:35.332 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:21:36.066 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:23:36.157 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:23:36.210 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:23:36.967 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:23:36.968 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:23:36.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:23:37.223 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:23:37.273 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:23:37.293 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T20:23:37.293 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T20:23:37.295 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T20:23:37.343 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:23:38.131 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:25:38.190 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:25:38.236 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:25:38.970 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:25:38.971 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:25:38.972 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:25:39.174 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:25:39.276 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:25:39.297 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T20:25:39.298 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T20:25:39.299 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T20:25:39.345 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:25:40.051 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:27:40.111 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:27:40.161 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:27:40.920 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:27:40.920 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:27:40.921 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:27:41.137 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:27:41.224 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:27:41.242 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T20:27:41.242 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T20:27:41.243 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T20:27:41.290 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:27:42.086 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:29:42.147 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:29:42.195 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:29:42.939 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:29:42.940 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:29:42.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:29:43.150 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:29:43.196 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:29:43.254 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T20:29:43.254 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T20:29:43.256 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T20:29:43.339 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:29:44.031 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:31:44.085 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:31:44.134 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:31:44.883 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:31:44.883 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:31:44.884 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:31:45.089 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:31:45.176 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:31:45.194 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T20:31:45.195 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T20:31:45.196 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T20:31:45.243 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:31:45.973 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:33:46.026 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:33:46.121 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:33:46.929 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:33:46.930 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:33:46.930 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:33:47.129 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:33:47.171 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:33:47.231 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T20:33:47.232 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T20:33:47.233 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T20:33:47.281 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:33:48.018 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:35:48.074 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:35:48.120 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:35:48.879 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:35:48.880 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:35:48.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:35:49.087 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:35:49.172 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:35:49.189 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T20:35:49.189 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T20:35:49.191 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T20:35:49.237 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:35:49.953 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:37:50.007 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:37:50.060 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:37:50.821 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:37:50.822 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:37:50.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:37:51.027 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:37:51.089 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:37:51.146 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T20:37:51.147 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T20:37:51.149 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T20:37:51.201 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:37:51.914 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:39:51.970 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:39:52.020 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:39:52.765 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:39:52.766 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:39:52.766 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:39:52.973 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:39:53.019 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:39:53.037 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T20:39:53.038 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T20:39:53.039 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T20:39:53.086 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:39:53.891 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:41:53.957 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:41:54.007 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:41:54.790 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:41:54.790 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:41:54.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:41:55.003 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:41:55.045 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:41:55.105 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T20:41:55.106 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T20:41:55.107 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T20:41:55.156 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:41:55.938 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:43:55.990 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:43:56.044 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:43:56.800 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:43:56.800 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:43:56.801 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:43:57.007 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:43:57.050 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:43:57.109 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T20:43:57.110 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T20:43:57.111 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T20:43:57.156 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:43:57.912 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:45:57.962 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:45:58.010 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:45:58.759 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:45:58.759 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:45:58.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:45:58.972 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:45:59.025 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:45:59.043 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T20:45:59.044 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T20:45:59.045 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T20:45:59.092 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:45:59.890 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:47:59.947 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:47:59.994 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:48:00.721 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:48:00.721 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:48:00.722 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:48:00.973 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:48:01.025 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:48:01.045 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T20:48:01.046 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T20:48:01.047 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T20:48:01.096 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:48:01.859 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:50:01.916 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:50:01.973 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:50:02.766 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:50:02.767 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:50:02.767 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:50:02.981 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:50:03.034 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:50:03.052 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T20:50:03.052 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T20:50:03.053 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T20:50:03.104 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:50:03.837 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:52:03.903 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:52:03.960 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:52:04.755 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:52:04.756 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:52:04.756 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:52:04.968 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:52:05.068 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:52:05.086 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T20:52:05.086 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T20:52:05.087 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T20:52:05.148 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:52:05.903 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:54:05.970 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:54:06.024 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:54:06.809 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:54:06.810 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:54:06.810 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:54:07.020 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:54:07.115 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:54:07.137 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T20:54:07.137 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T20:54:07.138 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T20:54:07.189 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:54:07.936 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:56:07.988 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:56:08.086 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:56:08.803 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:56:08.804 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:56:08.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:56:09.008 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:56:09.104 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:56:09.123 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T20:56:09.123 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T20:56:09.124 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T20:56:09.173 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:56:09.933 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T20:58:09.983 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T20:58:10.032 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:58:10.787 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T20:58:10.788 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:58:10.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T20:58:10.995 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:58:11.043 INFO:teuthology.misc:Compressing logs... 2023-01-14T20:58:11.100 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T20:58:11.100 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T20:58:11.102 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T20:58:11.146 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T20:58:11.869 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:00:11.922 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:00:11.971 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:00:12.858 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:00:12.859 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:00:12.860 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:00:13.063 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:00:13.152 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:00:13.169 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:00:13.169 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:00:13.171 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:00:13.217 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:00:13.955 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:02:14.016 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:02:14.064 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:02:14.846 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:02:14.847 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:02:14.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:02:15.053 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:02:15.096 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:02:15.114 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:02:15.114 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:02:15.115 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:02:15.167 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:02:15.885 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:04:15.945 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:04:15.992 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:04:16.771 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:04:16.772 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:04:16.772 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:04:16.974 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:04:17.017 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:04:17.035 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:04:17.035 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:04:17.037 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:04:17.086 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:04:17.812 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:06:17.864 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:06:17.913 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:06:18.712 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:06:18.713 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:06:18.714 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:06:18.922 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:06:19.015 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:06:19.032 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:06:19.033 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:06:19.034 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:06:19.081 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:06:19.828 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:08:19.880 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:08:19.928 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:08:20.685 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:08:20.685 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:08:20.686 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:08:20.889 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:08:20.933 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:08:20.951 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:08:20.951 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:08:20.952 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:08:21.001 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:08:21.750 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:10:21.804 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:10:21.858 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:10:22.619 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:10:22.620 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:10:22.621 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:10:22.822 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:10:22.875 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:10:22.934 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:10:22.935 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:10:22.937 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:10:23.009 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:10:23.765 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:12:23.825 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:12:23.870 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:12:24.633 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:12:24.633 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:12:24.634 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:12:24.889 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:12:24.936 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:12:24.993 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:12:24.993 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:12:24.995 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:12:25.039 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:12:25.730 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:14:25.793 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:14:25.848 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:14:26.692 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:14:26.693 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:14:26.694 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:14:26.903 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:14:26.991 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:14:27.009 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:14:27.009 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:14:27.011 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:14:27.056 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:14:27.811 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:16:27.864 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:16:27.909 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:16:28.665 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:16:28.666 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:16:28.666 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:16:28.872 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:16:28.916 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:16:28.934 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:16:28.934 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:16:28.935 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:16:28.981 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:16:29.684 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:18:29.747 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:18:29.836 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:18:30.552 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:18:30.553 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:18:30.553 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:18:30.761 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:18:30.847 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:18:30.863 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:18:30.864 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:18:30.865 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:18:30.908 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:18:31.618 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:20:31.669 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:20:31.715 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:20:32.460 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:20:32.460 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:20:32.461 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:20:32.666 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:20:32.710 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:20:32.726 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:20:32.726 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:20:32.727 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:20:32.773 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:20:33.492 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:22:33.546 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:22:33.596 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:22:34.303 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:22:34.303 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:22:34.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:22:34.519 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:22:34.566 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:22:34.629 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:22:34.629 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:22:34.631 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:22:34.676 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:22:35.414 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:24:35.472 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:24:35.523 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:24:36.273 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:24:36.274 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:24:36.274 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:24:36.482 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:24:36.524 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:24:36.540 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:24:36.541 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:24:36.542 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:24:36.589 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:24:37.288 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:26:37.340 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:26:37.387 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:26:38.124 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:26:38.125 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:26:38.125 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:26:38.323 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:26:38.367 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:26:38.426 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:26:38.426 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:26:38.427 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:26:38.475 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:26:39.214 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:28:39.274 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:28:39.321 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:28:40.011 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:28:40.011 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:28:40.012 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:28:40.221 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:28:40.307 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:28:40.326 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:28:40.326 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:28:40.327 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:28:40.372 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:28:41.078 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:30:41.133 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:30:41.175 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:30:41.884 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:30:41.884 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:30:41.885 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:30:42.096 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:30:42.187 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:30:42.206 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:30:42.206 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:30:42.208 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:30:42.254 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:30:42.913 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:32:42.967 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:32:43.015 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:32:43.712 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:32:43.712 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:32:43.713 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:32:43.918 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:32:44.013 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:32:44.031 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:32:44.031 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:32:44.033 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:32:44.076 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:32:44.730 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:34:44.786 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:34:44.834 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:34:45.571 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:34:45.572 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:34:45.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:34:45.774 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:34:45.816 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:34:45.832 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:34:45.832 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:34:45.834 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:34:45.881 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:34:46.636 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:36:46.690 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:36:46.735 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:36:47.488 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:36:47.489 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:36:47.489 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:36:47.696 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:36:47.783 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:36:47.800 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:36:47.800 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:36:47.802 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:36:47.847 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:36:48.579 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:38:48.642 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:38:48.691 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:38:49.434 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:38:49.435 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:38:49.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:38:49.636 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:38:49.723 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:38:49.742 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:38:49.742 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:38:49.744 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:38:49.791 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:38:50.501 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:40:50.557 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:40:50.648 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:40:51.414 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:40:51.414 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:40:51.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:40:51.628 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:40:51.671 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:40:51.731 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:40:51.731 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:40:51.733 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:40:51.777 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:40:52.495 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:42:52.567 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:42:52.614 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:42:53.375 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:42:53.376 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:42:53.377 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:42:53.627 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:42:53.676 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:42:53.730 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:42:53.730 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:42:53.732 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:42:53.777 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:42:54.456 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:44:54.508 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:44:54.555 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:44:55.349 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:44:55.349 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:44:55.350 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:44:55.561 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:44:55.667 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:44:55.685 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:44:55.685 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:44:55.687 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:44:55.735 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:44:56.437 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:46:56.510 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:46:56.559 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:46:57.352 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:46:57.353 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:46:57.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:46:57.558 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:46:57.648 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:46:57.667 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:46:57.667 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:46:57.668 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:46:57.713 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:46:58.393 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:48:58.452 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:48:58.502 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:48:59.239 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:48:59.239 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:48:59.240 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:48:59.456 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:48:59.547 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:48:59.566 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:48:59.566 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:48:59.568 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:48:59.614 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:49:00.292 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:51:00.347 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:51:00.392 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:51:01.128 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:51:01.129 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:51:01.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:51:01.336 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:51:01.383 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:51:01.400 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:51:01.400 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:51:01.402 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:51:01.442 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:51:02.123 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:53:02.182 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:53:02.230 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:53:02.971 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:53:02.972 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:53:02.972 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:53:03.224 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:53:03.312 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:53:03.329 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:53:03.330 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:53:03.331 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:53:03.380 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:53:04.180 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:55:04.236 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:55:04.286 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:55:05.053 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:55:05.054 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:55:05.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:55:05.269 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:55:05.352 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:55:05.368 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:55:05.369 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:55:05.370 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:55:05.417 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:55:06.163 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:57:06.213 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:57:06.256 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:57:07.027 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:57:07.028 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:57:07.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:57:07.239 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:57:07.327 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:57:07.345 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:57:07.345 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:57:07.346 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:57:07.392 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:57:08.159 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T21:59:08.215 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T21:59:08.267 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:59:09.019 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T21:59:09.020 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:59:09.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T21:59:09.227 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:59:09.317 INFO:teuthology.misc:Compressing logs... 2023-01-14T21:59:09.334 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T21:59:09.334 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T21:59:09.335 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T21:59:09.382 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T21:59:10.101 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:01:10.154 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:01:10.199 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:01:10.945 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:01:10.945 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:01:10.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:01:11.172 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:01:11.264 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:01:11.280 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:01:11.281 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:01:11.282 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:01:11.327 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:01:12.048 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:03:12.099 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:03:12.188 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:03:12.951 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:03:12.952 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:03:12.952 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:03:13.204 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:03:13.244 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:03:13.259 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:03:13.260 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:03:13.261 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:03:13.305 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:03:13.989 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:05:14.040 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:05:14.086 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:05:14.790 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:05:14.791 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:05:14.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:05:15.000 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:05:15.098 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:05:15.118 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:05:15.118 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:05:15.120 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:05:15.162 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:05:15.936 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:07:16.004 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:07:16.055 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:07:16.841 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:07:16.842 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:07:16.842 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:07:17.050 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:07:17.153 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:07:17.173 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:07:17.173 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:07:17.175 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:07:17.219 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:07:17.906 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:09:17.960 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:09:18.004 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:09:18.685 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:09:18.686 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:09:18.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:09:18.893 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:09:18.952 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:09:19.016 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:09:19.016 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:09:19.018 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:09:19.062 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:09:19.826 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:11:19.878 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:11:19.924 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:11:20.623 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:11:20.624 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:11:20.625 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:11:20.825 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:11:20.917 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:11:20.933 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:11:20.933 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:11:20.935 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:11:20.981 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:11:21.636 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:13:21.702 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:13:21.750 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:13:22.441 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:13:22.442 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:13:22.442 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:13:22.658 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:13:22.747 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:13:22.766 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:13:22.767 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:13:22.768 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:13:22.816 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:13:23.498 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:15:23.556 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:15:23.605 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:15:24.297 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:15:24.298 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:15:24.299 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:15:24.503 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:15:24.550 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:15:24.569 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:15:24.569 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:15:24.570 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:15:24.615 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:15:25.317 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:17:25.383 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:17:25.434 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:17:26.184 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:17:26.185 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:17:26.185 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:17:26.388 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:17:26.435 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:17:26.493 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:17:26.493 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:17:26.495 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:17:26.540 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:17:27.275 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:19:27.336 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:19:27.378 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:19:28.085 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:19:28.085 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:19:28.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:19:28.284 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:19:28.371 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:19:28.388 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:19:28.389 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:19:28.390 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:19:28.436 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:19:29.093 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:21:29.148 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:21:29.198 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:21:30.007 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:21:30.008 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:21:30.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:21:30.213 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:21:30.261 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:21:30.277 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:21:30.277 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:21:30.279 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:21:30.322 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:21:31.031 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:23:31.091 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:23:31.139 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:23:31.894 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:23:31.895 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:23:31.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:23:32.110 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:23:32.151 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:23:32.167 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:23:32.168 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:23:32.169 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:23:32.216 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:23:32.975 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:25:33.025 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:25:33.111 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:25:33.867 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:25:33.867 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:25:33.868 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:25:34.081 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:25:34.170 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:25:34.187 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:25:34.188 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:25:34.189 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:25:34.235 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:25:34.987 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:27:35.049 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:27:35.095 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:27:35.852 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:27:35.853 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:27:35.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:27:36.060 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:27:36.102 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:27:36.119 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:27:36.119 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:27:36.120 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:27:36.166 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:27:36.888 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:29:36.945 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:29:36.994 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:29:37.788 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:29:37.789 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:29:37.789 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:29:37.996 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:29:38.046 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:29:38.104 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:29:38.105 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:29:38.106 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:29:38.155 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:29:38.855 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:31:38.917 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:31:38.964 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:31:39.767 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:31:39.768 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:31:39.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:31:39.969 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:31:40.062 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:31:40.080 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:31:40.080 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:31:40.082 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:31:40.125 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:31:40.849 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:33:40.907 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:33:40.951 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:33:41.681 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:33:41.682 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:33:41.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:33:41.878 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:33:41.974 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:33:41.992 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:33:41.992 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:33:41.994 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:33:42.037 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:33:42.702 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:35:42.756 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:35:42.805 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:35:43.611 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:35:43.611 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:35:43.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:35:43.818 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:35:43.860 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:35:43.878 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:35:43.879 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:35:43.880 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:35:43.926 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:35:44.613 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:37:44.671 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:37:44.717 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:37:45.402 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:37:45.403 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:37:45.403 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:37:45.613 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:37:45.659 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:37:45.678 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:37:45.679 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:37:45.680 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:37:45.726 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:37:46.421 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:39:46.479 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:39:46.523 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:39:47.241 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:39:47.241 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:39:47.242 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:39:47.443 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:39:47.534 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:39:47.551 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:39:47.551 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:39:47.552 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:39:47.598 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:39:48.333 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:41:48.384 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:41:48.431 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:41:49.167 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:41:49.168 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:41:49.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:41:49.376 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:41:49.423 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:41:49.440 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:41:49.440 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:41:49.442 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:41:49.487 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:41:50.187 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:43:50.235 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:43:50.284 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:43:51.038 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:43:51.039 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:43:51.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:43:51.297 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:43:51.394 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:43:51.411 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:43:51.411 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:43:51.413 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:43:51.461 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:43:52.141 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:45:52.192 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:45:52.239 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:45:52.978 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:45:52.979 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:45:52.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:45:53.194 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:45:53.287 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:45:53.305 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:45:53.306 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:45:53.308 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:45:53.354 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:45:54.060 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:47:54.113 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:47:54.204 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:47:54.964 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:47:54.964 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:47:54.965 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:47:55.172 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:47:55.267 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:47:55.284 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:47:55.285 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:47:55.286 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:47:55.330 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:47:55.999 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:49:56.070 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:49:56.118 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:49:56.871 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:49:56.871 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:49:56.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:49:57.068 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:49:57.111 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:49:57.129 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:49:57.129 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:49:57.131 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:49:57.175 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:49:57.929 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:51:57.983 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:51:58.031 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:51:58.788 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:51:58.789 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:51:58.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:51:59.001 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:51:59.094 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:51:59.114 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:51:59.114 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:51:59.116 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:51:59.161 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:51:59.845 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:53:59.898 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:53:59.948 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:54:00.727 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:54:00.728 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:54:00.728 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:54:00.934 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:54:01.028 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:54:01.045 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:54:01.046 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:54:01.048 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:54:01.092 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:54:01.796 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:56:01.855 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:56:01.906 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:56:02.717 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:56:02.718 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:56:02.718 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:56:02.927 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:56:02.982 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:56:03.033 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:56:03.033 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:56:03.035 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:56:03.079 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:56:03.780 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T22:58:03.837 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:58:03.883 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:58:04.585 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T22:58:04.586 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:58:04.586 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T22:58:04.796 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:58:04.855 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:58:04.874 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T22:58:04.874 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T22:58:04.877 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T22:58:04.927 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T22:58:05.697 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:00:05.758 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:00:05.806 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:00:06.562 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:00:06.562 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:00:06.563 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:00:06.767 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:00:06.859 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:00:06.876 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:00:06.877 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:00:06.878 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:00:06.918 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:00:07.673 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:02:07.726 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:02:07.774 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:02:08.512 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:02:08.512 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:02:08.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:02:08.723 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:02:08.775 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:02:08.833 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:02:08.833 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:02:08.834 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:02:08.879 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:02:09.577 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:04:09.628 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:04:09.677 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:04:10.428 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:04:10.429 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:04:10.429 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:04:10.636 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:04:10.730 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:04:10.747 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:04:10.748 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:04:10.749 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:04:10.793 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:04:11.521 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:06:11.574 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:06:11.624 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:06:12.321 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:06:12.321 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:06:12.322 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:06:12.525 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:06:12.566 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:06:12.584 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:06:12.584 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:06:12.585 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:06:12.633 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:06:13.326 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:08:13.377 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:08:13.421 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:08:14.122 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:08:14.123 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:08:14.123 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:08:14.369 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:08:14.422 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:08:14.439 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:08:14.439 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:08:14.440 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:08:14.489 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:08:15.170 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:10:15.226 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:10:15.310 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:10:16.032 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:10:16.032 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:10:16.033 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:10:16.229 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:10:16.275 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:10:16.293 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:10:16.294 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:10:16.295 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:10:16.341 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:10:17.023 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:12:17.085 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:12:17.145 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:12:17.889 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:12:17.889 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:12:17.890 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:12:18.098 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:12:18.187 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:12:18.205 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:12:18.205 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:12:18.207 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:12:18.252 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:12:18.959 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:14:19.009 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:14:19.054 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:14:19.768 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:14:19.769 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:14:19.769 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:14:20.018 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:14:20.059 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:14:20.117 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:14:20.117 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:14:20.118 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:14:20.162 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:14:20.914 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:16:20.968 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:16:21.014 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:16:21.722 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:16:21.723 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:16:21.723 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:16:21.931 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:16:22.023 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:16:22.040 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:16:22.040 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:16:22.042 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:16:22.088 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:16:22.778 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:18:22.828 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:18:22.874 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:18:23.579 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:18:23.580 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:18:23.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:18:23.776 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:18:23.867 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:18:23.885 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:18:23.885 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:18:23.887 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:18:23.931 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:18:24.615 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:20:24.668 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:20:24.715 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:20:25.431 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:20:25.432 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:20:25.432 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:20:25.633 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:20:25.675 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:20:25.733 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:20:25.734 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:20:25.735 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:20:25.777 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:20:26.470 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:22:26.529 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:22:26.575 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:22:27.350 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:22:27.350 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:22:27.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:22:27.558 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:22:27.606 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:22:27.624 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:22:27.624 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:22:27.626 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:22:27.672 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:22:28.395 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:24:28.448 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:24:28.496 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:24:29.268 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:24:29.268 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:24:29.269 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:24:29.477 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:24:29.523 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:24:29.541 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:24:29.541 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:24:29.543 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:24:29.589 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:24:30.343 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:26:30.403 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:26:30.451 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:26:31.276 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:26:31.277 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:26:31.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:26:31.477 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:26:31.522 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:26:31.581 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:26:31.582 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:26:31.583 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:26:31.629 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:26:32.348 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:28:32.405 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:28:32.454 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:28:33.180 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:28:33.181 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:28:33.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:28:33.393 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:28:33.438 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:28:33.454 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:28:33.454 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:28:33.456 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:28:33.499 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:28:34.161 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:30:34.224 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:30:34.271 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:30:35.007 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:30:35.008 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:30:35.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:30:35.204 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:30:35.245 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:30:35.305 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:30:35.306 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:30:35.308 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:30:35.353 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:30:36.083 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:32:36.138 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:32:36.227 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:32:37.003 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:32:37.004 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:32:37.004 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:32:37.213 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:32:37.258 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:32:37.276 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:32:37.276 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:32:37.277 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:32:37.328 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:32:38.030 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:34:38.080 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:34:38.128 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:34:38.828 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:34:38.829 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:34:38.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:34:39.030 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:34:39.121 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:34:39.138 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:34:39.138 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:34:39.140 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:34:39.184 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:34:39.889 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:36:39.938 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:36:39.984 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:36:40.762 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:36:40.763 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:36:40.763 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:36:40.967 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:36:41.058 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:36:41.076 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:36:41.077 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:36:41.078 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:36:41.126 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:36:41.847 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:38:41.901 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:38:41.950 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:38:42.668 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:38:42.669 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:38:42.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:38:42.875 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:38:42.957 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:38:42.974 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:38:42.975 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:38:42.976 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:38:43.020 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:38:43.699 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:40:43.749 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:40:43.797 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:40:44.609 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:40:44.610 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:40:44.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:40:44.821 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:40:44.865 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:40:44.883 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:40:44.883 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:40:44.885 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:40:44.941 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:40:45.644 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:42:45.713 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:42:45.762 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:42:46.560 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:42:46.561 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:42:46.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:42:46.814 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:42:46.861 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:42:46.878 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:42:46.878 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:42:46.879 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:42:46.949 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:42:47.662 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:44:47.718 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:44:47.764 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:44:48.550 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:44:48.551 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:44:48.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:44:48.756 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:44:48.853 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:44:48.870 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:44:48.870 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:44:48.872 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:44:48.913 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:44:49.626 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:46:49.676 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:46:49.721 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:46:50.438 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:46:50.439 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:46:50.439 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:46:50.647 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:46:50.739 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:46:50.756 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:46:50.757 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:46:50.759 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:46:50.832 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:46:51.526 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:48:51.580 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:48:51.629 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:48:52.335 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:48:52.336 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:48:52.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:48:52.547 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:48:52.638 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:48:52.654 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:48:52.654 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:48:52.657 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:48:52.715 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:48:53.429 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:50:53.484 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:50:53.539 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:50:54.327 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:50:54.328 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:50:54.329 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:50:54.536 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:50:54.577 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:50:54.595 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:50:54.595 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:50:54.596 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:50:54.641 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:50:55.395 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:52:55.447 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:52:55.495 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:52:56.241 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:52:56.242 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:52:56.242 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:52:56.446 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:52:56.489 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:52:56.504 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:52:56.504 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:52:56.506 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:52:56.563 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:52:57.283 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:54:57.336 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:54:57.457 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:54:58.167 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:54:58.167 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:54:58.168 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:54:58.375 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:54:58.464 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:54:58.479 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:54:58.479 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:54:58.481 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:54:58.528 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:54:59.271 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:56:59.326 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:56:59.372 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:57:00.102 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:57:00.103 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:57:00.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:57:00.309 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:57:00.405 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:57:00.421 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:57:00.422 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:57:00.423 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:57:00.467 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:57:01.150 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:59:01.210 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:59:01.258 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:59:02.041 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-14T23:59:02.041 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:59:02.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-14T23:59:02.249 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:59:02.344 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:59:02.360 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:59:02.360 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:59:02.362 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:59:02.403 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-14T23:59:03.109 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:01:03.165 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:01:03.212 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:01:03.938 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:01:03.939 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:01:03.939 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:01:04.139 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:01:04.191 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:01:04.210 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:01:04.210 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:01:04.211 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:01:04.257 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:01:05.007 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:03:05.065 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:03:05.113 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:03:05.830 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:03:05.830 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:03:05.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:03:06.029 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:03:06.084 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:03:06.140 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:03:06.141 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:03:06.142 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:03:06.186 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:03:06.875 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:05:06.932 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:05:06.982 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:05:07.702 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:05:07.703 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:05:07.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:05:07.909 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:05:07.956 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:05:08.016 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:05:08.016 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:05:08.018 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:05:08.062 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:05:08.729 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:07:08.784 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:07:08.828 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:07:09.565 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:07:09.566 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:07:09.566 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:07:09.773 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:07:09.817 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:07:09.876 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:07:09.876 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:07:09.878 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:07:09.921 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:07:10.605 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:09:10.658 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:09:10.703 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:09:11.405 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:09:11.406 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:09:11.406 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:09:11.609 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:09:11.701 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:09:11.720 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:09:11.721 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:09:11.722 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:09:11.766 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:09:12.425 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:11:12.475 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:11:12.524 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:11:13.187 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:11:13.188 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:11:13.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:11:13.390 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:11:13.487 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:11:13.505 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:11:13.505 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:11:13.506 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:11:13.550 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:11:14.230 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:13:14.281 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:13:14.328 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:13:15.040 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:13:15.041 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:13:15.041 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:13:15.250 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:13:15.294 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:13:15.352 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:13:15.352 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:13:15.354 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:13:15.397 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:13:16.052 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:15:16.116 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:15:16.164 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:15:16.866 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:15:16.866 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:15:16.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:15:17.074 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:15:17.162 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:15:17.180 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:15:17.180 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:15:17.182 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:15:17.228 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:15:17.937 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:17:17.990 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:17:18.074 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:17:18.796 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:17:18.797 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:17:18.797 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:17:19.007 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:17:19.103 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:17:19.123 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:17:19.124 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:17:19.125 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:17:19.170 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:17:19.919 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:19:19.978 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:19:20.026 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:19:20.787 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:19:20.788 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:19:20.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:19:21.029 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:19:21.074 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:19:21.120 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:19:21.120 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:19:21.122 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:19:21.167 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:19:21.902 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:21:21.969 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:21:22.019 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:21:22.720 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:21:22.721 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:21:22.721 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:21:22.932 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:21:22.973 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:21:22.990 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:21:22.990 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:21:22.991 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:21:23.037 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:21:23.707 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:23:23.761 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:23:23.809 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:23:24.520 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:23:24.520 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:23:24.521 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:23:24.727 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:23:24.817 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:23:24.834 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:23:24.834 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:23:24.836 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:23:24.882 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:23:25.554 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:25:25.614 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:25:25.666 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:25:26.511 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:25:26.512 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:25:26.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:25:26.723 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:25:26.771 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:25:26.789 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:25:26.789 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:25:26.791 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:25:26.834 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:25:27.546 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:27:27.597 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:27:27.642 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:27:28.339 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:27:28.340 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:27:28.340 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:27:28.594 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:27:28.647 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:27:28.667 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:27:28.667 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:27:28.668 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:27:28.713 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:27:29.390 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:29:29.442 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:29:29.489 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:29:30.210 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:29:30.211 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:29:30.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:29:30.424 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:29:30.467 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:29:30.485 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:29:30.485 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:29:30.487 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:29:30.532 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:29:31.204 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:31:31.256 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:31:31.302 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:31:32.027 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:31:32.027 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:31:32.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:31:32.229 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:31:32.281 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:31:32.299 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:31:32.300 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:31:32.301 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:31:32.346 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:31:33.010 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:33:33.073 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:33:33.118 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:33:33.862 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:33:33.862 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:33:33.863 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:33:34.070 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:33:34.164 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:33:34.181 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:33:34.181 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:33:34.182 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:33:34.225 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:33:34.938 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:35:34.991 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:35:35.042 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:35:35.749 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:35:35.750 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:35:35.750 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:35:35.957 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:35:36.000 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:35:36.018 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:35:36.018 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:35:36.019 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:35:36.062 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:35:36.741 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:37:36.807 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:37:36.853 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:37:37.523 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:37:37.523 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:37:37.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:37:37.740 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:37:37.831 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:37:37.850 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:37:37.850 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:37:37.852 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:37:37.892 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:37:38.588 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:39:38.642 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:39:38.734 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:39:39.446 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:39:39.446 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:39:39.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:39:39.650 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:39:39.695 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:39:39.712 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:39:39.713 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:39:39.714 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:39:39.764 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:39:40.462 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:41:40.522 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:41:40.568 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:41:41.280 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:41:41.281 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:41:41.281 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:41:41.486 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:41:41.570 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:41:41.588 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:41:41.588 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:41:41.590 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:41:41.634 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:41:42.314 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:43:42.374 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:43:42.418 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:43:43.120 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:43:43.120 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:43:43.121 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:43:43.325 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:43:43.414 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:43:43.432 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:43:43.432 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:43:43.433 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:43:43.478 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:43:44.132 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:45:44.187 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:45:44.239 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:45:44.981 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:45:44.982 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:45:44.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:45:45.186 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:45:45.232 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:45:45.292 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:45:45.292 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:45:45.294 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:45:45.340 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:45:46.020 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:47:46.073 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:47:46.130 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:47:46.859 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:47:46.859 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:47:46.860 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:47:47.056 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:47:47.103 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:47:47.120 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:47:47.120 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:47:47.122 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:47:47.174 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:47:47.866 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:49:47.919 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:49:47.967 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:49:48.686 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:49:48.687 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:49:48.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:49:48.938 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:49:48.981 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:49:49.040 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:49:49.040 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:49:49.042 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:49:49.085 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:49:49.765 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:51:49.823 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:51:49.874 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:51:50.583 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:51:50.584 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:51:50.585 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:51:50.838 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:51:50.882 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:51:50.940 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:51:50.940 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:51:50.942 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:51:50.989 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:51:51.685 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:53:51.737 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:53:51.782 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:53:52.509 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:53:52.509 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:53:52.510 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:53:52.718 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:53:52.811 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:53:52.829 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:53:52.830 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:53:52.831 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:53:52.877 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:53:53.554 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:55:53.605 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:55:53.650 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:55:54.336 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:55:54.337 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:55:54.337 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:55:54.537 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:55:54.627 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:55:54.645 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:55:54.646 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:55:54.647 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:55:54.693 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:55:55.414 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:57:55.466 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:57:55.513 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:57:56.385 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:57:56.385 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:57:56.386 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:57:56.596 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:57:56.640 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:57:56.700 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:57:56.700 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:57:56.702 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:57:56.746 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:57:57.470 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:59:57.536 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:59:57.580 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:59:58.306 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T00:59:58.307 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:59:58.307 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T00:59:58.550 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:59:58.596 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:59:58.613 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:59:58.614 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:59:58.615 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:59:58.659 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T00:59:59.357 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:01:59.405 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:01:59.501 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:02:00.343 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:02:00.344 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:02:00.344 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:02:00.547 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:02:00.600 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:02:00.618 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:02:00.619 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:02:00.620 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:02:00.665 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:02:01.337 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:04:01.392 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:04:01.436 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:04:02.161 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:04:02.162 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:04:02.162 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:04:02.366 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:04:02.462 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:04:02.480 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:04:02.481 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:04:02.482 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:04:02.548 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:04:03.246 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:06:03.300 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:06:03.349 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:06:04.100 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:06:04.101 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:06:04.101 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:06:04.304 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:06:04.350 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:06:04.366 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:06:04.367 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:06:04.368 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:06:04.420 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:06:05.146 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:08:05.201 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:08:05.248 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:08:06.047 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:08:06.047 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:08:06.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:08:06.259 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:08:06.307 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:08:06.365 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:08:06.366 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:08:06.367 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:08:06.414 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:08:07.134 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:10:07.197 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:10:07.241 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:10:07.954 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:10:07.954 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:10:07.955 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:10:08.160 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:10:08.265 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:10:08.283 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:10:08.284 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:10:08.285 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:10:08.329 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:10:09.031 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:12:09.103 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:12:09.149 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:12:09.891 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:12:09.892 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:12:09.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:12:10.100 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:12:10.205 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:12:10.224 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:12:10.224 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:12:10.226 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:12:10.269 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:12:10.991 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:14:11.053 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:14:11.105 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:14:11.848 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:14:11.849 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:14:11.849 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:14:12.061 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:14:12.154 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:14:12.171 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:14:12.172 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:14:12.173 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:14:12.216 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:14:12.889 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:16:12.943 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:16:12.990 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:16:13.768 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:16:13.768 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:16:13.769 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:16:14.020 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:16:14.065 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:16:14.125 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:16:14.125 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:16:14.126 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:16:14.170 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:16:14.869 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:18:14.922 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:18:14.969 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:18:15.712 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:18:15.712 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:18:15.713 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:18:15.970 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:18:16.009 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:18:16.069 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:18:16.069 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:18:16.071 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:18:16.118 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:18:16.846 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:20:16.912 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:20:16.962 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:20:17.740 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:20:17.741 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:20:17.741 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:20:17.955 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:20:17.999 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:20:18.017 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:20:18.017 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:20:18.019 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:20:18.066 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:20:18.770 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:22:18.828 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:22:18.876 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:22:19.602 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:22:19.602 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:22:19.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:22:19.813 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:22:19.861 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:22:19.878 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:22:19.878 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:22:19.880 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:22:19.924 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:22:20.633 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:24:20.688 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:24:20.783 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:24:21.576 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:24:21.577 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:24:21.577 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:24:21.776 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:24:21.818 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:24:21.837 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:24:21.838 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:24:21.839 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:24:21.887 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:24:22.605 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:26:22.658 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:26:22.711 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:26:23.454 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:26:23.455 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:26:23.455 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:26:23.657 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:26:23.701 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:26:23.718 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:26:23.718 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:26:23.719 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:26:23.769 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:26:24.454 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:28:24.509 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:28:24.557 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:28:25.307 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:28:25.308 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:28:25.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:28:25.518 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:28:25.563 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:28:25.581 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:28:25.581 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:28:25.582 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:28:25.634 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:28:26.402 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:30:26.455 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:30:26.498 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:30:27.243 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:30:27.243 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:30:27.244 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:30:27.459 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:30:27.506 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:30:27.523 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:30:27.523 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:30:27.525 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:30:27.572 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:30:28.286 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:32:28.338 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:32:28.388 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:32:29.076 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:32:29.077 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:32:29.077 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:32:29.330 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:32:29.376 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:32:29.425 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:32:29.425 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:32:29.426 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:32:29.471 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:32:30.143 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:34:30.201 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:34:30.245 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:34:30.941 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:34:30.942 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:34:30.942 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:34:31.139 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:34:31.182 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:34:31.240 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:34:31.241 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:34:31.242 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:34:31.285 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:34:31.943 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:36:31.994 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:36:32.040 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:36:32.739 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:36:32.740 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:36:32.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:36:32.951 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:36:33.040 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:36:33.059 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:36:33.059 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:36:33.060 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:36:33.107 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:36:33.759 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:38:33.825 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:38:33.870 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:38:34.586 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:38:34.586 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:38:34.587 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:38:34.791 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:38:34.837 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:38:34.896 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:38:34.897 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:38:34.898 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:38:34.942 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:38:35.630 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:40:35.684 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:40:35.731 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:40:36.463 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:40:36.464 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:40:36.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:40:36.665 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:40:36.704 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:40:36.722 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:40:36.722 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:40:36.724 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:40:36.766 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:40:37.462 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:42:37.513 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:42:37.557 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:42:38.286 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:42:38.287 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:42:38.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:42:38.488 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:42:38.535 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:42:38.594 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:42:38.594 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:42:38.595 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:42:38.636 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:42:39.349 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:44:39.400 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:44:39.446 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:44:40.196 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:44:40.197 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:44:40.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:44:40.408 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:44:40.495 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:44:40.513 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:44:40.513 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:44:40.515 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:44:40.560 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:44:41.214 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:46:41.266 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:46:41.340 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:46:42.040 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:46:42.040 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:46:42.041 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:46:42.294 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:46:42.337 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:46:42.355 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:46:42.355 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:46:42.357 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:46:42.401 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:46:43.049 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:48:43.100 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:48:43.143 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:48:43.817 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:48:43.818 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:48:43.818 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:48:44.027 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:48:44.119 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:48:44.137 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:48:44.137 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:48:44.138 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:48:44.185 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:48:44.837 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:50:44.897 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:50:44.944 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:50:45.615 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:50:45.616 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:50:45.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:50:45.816 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:50:45.907 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:50:45.925 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:50:45.925 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:50:45.927 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:50:45.971 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:50:46.652 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:52:46.704 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:52:46.748 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:52:47.447 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:52:47.447 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:52:47.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:52:47.648 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:52:47.740 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:52:47.758 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:52:47.758 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:52:47.760 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:52:47.801 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:52:48.481 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:54:48.549 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:54:48.598 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:54:49.308 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:54:49.308 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:54:49.309 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:54:49.516 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:54:49.558 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:54:49.616 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:54:49.616 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:54:49.618 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:54:49.661 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:54:50.311 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:56:50.363 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:56:50.409 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:56:51.139 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:56:51.140 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:56:51.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:56:51.353 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:56:51.394 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:56:51.411 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:56:51.411 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:56:51.412 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:56:51.455 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:56:52.148 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:58:52.221 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:58:52.271 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:58:53.024 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T01:58:53.025 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:58:53.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T01:58:53.233 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:58:53.326 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:58:53.343 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:58:53.344 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:58:53.345 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:58:53.389 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T01:58:54.065 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:00:54.121 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:00:54.167 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:00:54.866 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:00:54.866 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:00:54.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:00:55.073 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:00:55.118 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:00:55.135 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:00:55.136 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:00:55.137 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:00:55.178 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:00:55.835 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:02:55.890 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:02:55.936 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:02:56.647 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:02:56.647 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:02:56.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:02:56.853 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:02:56.899 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:02:56.917 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:02:56.917 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:02:56.919 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:02:56.960 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:02:57.659 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:04:57.718 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:04:57.760 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:04:58.493 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:04:58.493 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:04:58.493 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:04:58.700 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:04:58.741 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:04:58.758 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:04:58.758 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:04:58.760 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:04:58.806 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:04:59.520 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:06:59.572 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:06:59.616 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:07:00.292 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:07:00.293 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:07:00.293 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:07:00.508 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:07:00.597 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:07:00.616 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:07:00.616 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:07:00.617 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:07:00.661 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:07:01.385 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:09:01.450 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:09:01.527 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:09:02.229 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:09:02.229 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:09:02.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:09:02.429 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:09:02.470 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:09:02.529 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:09:02.530 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:09:02.531 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:09:02.586 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:09:03.282 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:11:03.332 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:11:03.376 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:11:04.084 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:11:04.085 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:11:04.085 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:11:04.291 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:11:04.385 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:11:04.401 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:11:04.402 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:11:04.404 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:11:04.445 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:11:05.157 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:13:05.213 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:13:05.259 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:13:06.069 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:13:06.070 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:13:06.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:13:06.278 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:13:06.370 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:13:06.387 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:13:06.388 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:13:06.389 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:13:06.429 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:13:07.138 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:15:07.193 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:15:07.240 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:15:07.937 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:15:07.938 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:15:07.938 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:15:08.137 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:15:08.181 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:15:08.198 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:15:08.199 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:15:08.200 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:15:08.245 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:15:08.918 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:17:08.975 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:17:09.023 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:17:09.791 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:17:09.792 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:17:09.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:17:10.000 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:17:10.094 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:17:10.112 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:17:10.112 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:17:10.114 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:17:10.154 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:17:10.841 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:19:10.901 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:19:10.943 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:19:11.664 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:19:11.665 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:19:11.665 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:19:11.867 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:19:11.954 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:19:11.971 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:19:11.971 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:19:11.974 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:19:12.016 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:19:12.727 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:21:12.784 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:21:12.831 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:21:13.614 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:21:13.615 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:21:13.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:21:13.823 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:21:13.869 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:21:13.885 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:21:13.885 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:21:13.887 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:21:13.928 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:21:14.590 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:23:14.651 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:23:14.698 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:23:15.443 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:23:15.443 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:23:15.443 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:23:15.644 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:23:15.689 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:23:15.706 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:23:15.706 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:23:15.708 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:23:15.753 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:23:16.447 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:25:16.518 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:25:16.565 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:25:17.297 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:25:17.298 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:25:17.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:25:17.504 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:25:17.551 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:25:17.569 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:25:17.569 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:25:17.570 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:25:17.611 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:25:18.293 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:27:18.346 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:27:18.394 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:27:19.098 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:27:19.099 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:27:19.099 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:27:19.302 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:27:19.389 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:27:19.407 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:27:19.408 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:27:19.409 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:27:19.454 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:27:20.133 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:29:20.200 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:29:20.249 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:29:20.951 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:29:20.952 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:29:20.952 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:29:21.157 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:29:21.199 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:29:21.216 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:29:21.216 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:29:21.217 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:29:21.260 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:29:21.995 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:31:22.048 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:31:22.134 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:31:22.869 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:31:22.870 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:31:22.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:31:23.076 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:31:23.167 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:31:23.184 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:31:23.185 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:31:23.186 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:31:23.234 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:31:23.944 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:33:23.999 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:33:24.045 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:33:24.759 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:33:24.760 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:33:24.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:33:24.962 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:33:25.013 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:33:25.073 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:33:25.073 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:33:25.075 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:33:25.119 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:33:25.811 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:35:25.870 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:35:25.914 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:35:26.617 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:35:26.617 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:35:26.618 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:35:26.821 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:35:26.866 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:35:26.884 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:35:26.884 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:35:26.885 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:35:26.926 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:35:27.614 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:37:27.673 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:37:27.719 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:37:28.420 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:37:28.421 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:37:28.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:37:28.625 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:37:28.674 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:37:28.732 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:37:28.733 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:37:28.734 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:37:28.777 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:37:29.469 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:39:29.522 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:39:29.569 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:39:30.303 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:39:30.304 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:39:30.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:39:30.509 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:39:30.551 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:39:30.567 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:39:30.568 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:39:30.570 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:39:30.611 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:39:31.276 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:41:31.325 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:41:31.368 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:41:32.062 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:41:32.063 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:41:32.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:41:32.255 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:41:32.301 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:41:32.318 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:41:32.319 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:41:32.320 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:41:32.362 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:41:32.997 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:43:33.054 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:43:33.099 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:43:33.789 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:43:33.790 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:43:33.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:43:33.997 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:43:34.043 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:43:34.061 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:43:34.061 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:43:34.063 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:43:34.105 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:43:34.786 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:45:34.837 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:45:34.882 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:45:35.607 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:45:35.608 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:45:35.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:45:35.817 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:45:35.867 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:45:35.885 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:45:35.885 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:45:35.887 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:45:35.928 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:45:36.589 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:47:36.642 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:47:36.687 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:47:37.416 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:47:37.416 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:47:37.417 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:47:37.616 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:47:37.704 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:47:37.720 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:47:37.720 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:47:37.722 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:47:37.769 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:47:38.463 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:49:38.522 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:49:38.568 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:49:39.317 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:49:39.317 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:49:39.318 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:49:39.525 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:49:39.567 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:49:39.624 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:49:39.625 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:49:39.626 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:49:39.670 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:49:40.344 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:51:40.409 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:51:40.454 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:51:41.147 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:51:41.147 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:51:41.148 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:51:41.348 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:51:41.390 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:51:41.407 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:51:41.407 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:51:41.409 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:51:41.449 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:51:42.098 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:53:42.164 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:53:42.247 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:53:42.954 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:53:42.955 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:53:42.955 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:53:43.163 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:53:43.204 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:53:43.221 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:53:43.222 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:53:43.223 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:53:43.266 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:53:43.937 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:55:43.996 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:55:44.040 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:55:44.762 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:55:44.762 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:55:44.763 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:55:44.961 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:55:45.003 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:55:45.061 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:55:45.061 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:55:45.064 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:55:45.109 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:55:45.833 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:57:45.897 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:57:45.946 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:57:46.683 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:57:46.684 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:57:46.685 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:57:46.887 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:57:46.929 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:57:46.946 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:57:46.946 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:57:46.947 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:57:46.989 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:57:47.644 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:59:47.694 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:59:47.736 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:59:48.432 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T02:59:48.433 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:59:48.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T02:59:48.643 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:59:48.731 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:59:48.748 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:59:48.748 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:59:48.749 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:59:48.795 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T02:59:49.535 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:01:49.586 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:01:49.634 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:01:50.397 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:01:50.398 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:01:50.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:01:50.605 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:01:50.648 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:01:50.666 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:01:50.666 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:01:50.667 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:01:50.712 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:01:51.449 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:03:51.505 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:03:51.549 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:03:52.232 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:03:52.233 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:03:52.233 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:03:52.436 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:03:52.481 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:03:52.497 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:03:52.497 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:03:52.498 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:03:52.541 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:03:53.214 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:05:53.265 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:05:53.310 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:05:54.064 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:05:54.065 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:05:54.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:05:54.269 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:05:54.360 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:05:54.377 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:05:54.377 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:05:54.378 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:05:54.421 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:05:55.102 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:07:55.162 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:07:55.205 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:07:55.943 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:07:55.944 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:07:55.944 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:07:56.150 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:07:56.241 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:07:56.258 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:07:56.258 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:07:56.260 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:07:56.305 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:07:56.984 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:09:57.037 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:09:57.080 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:09:57.764 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:09:57.764 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:09:57.765 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:09:57.962 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:09:58.007 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:09:58.065 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:09:58.065 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:09:58.066 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:09:58.112 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:09:58.822 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:11:58.876 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:11:58.922 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:11:59.640 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:11:59.641 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:11:59.641 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:11:59.849 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:11:59.943 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:11:59.961 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:11:59.962 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:11:59.963 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:12:00.005 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:12:00.653 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:14:00.714 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:14:00.757 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:14:01.443 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:14:01.444 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:14:01.444 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:14:01.643 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:14:01.684 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:14:01.700 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:14:01.700 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:14:01.701 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:14:01.743 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:14:02.440 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:16:02.493 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:16:02.569 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:16:03.262 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:16:03.262 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:16:03.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:16:03.460 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:16:03.503 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:16:03.520 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:16:03.521 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:16:03.522 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:16:03.565 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:16:04.228 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:18:04.282 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:18:04.329 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:18:05.025 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:18:05.026 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:18:05.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:18:05.229 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:18:05.278 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:18:05.293 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:18:05.294 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:18:05.295 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:18:05.338 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:18:06.040 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:20:06.115 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:20:06.162 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:20:06.870 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:20:06.870 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:20:06.871 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:20:07.074 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:20:07.166 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:20:07.185 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:20:07.185 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:20:07.187 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:20:07.228 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:20:07.932 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:22:07.983 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:22:08.029 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:22:08.764 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:22:08.765 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:22:08.765 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:22:08.976 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:22:09.069 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:22:09.085 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:22:09.085 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:22:09.087 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:22:09.135 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:22:09.790 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:24:09.844 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:24:09.889 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:24:10.576 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:24:10.577 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:24:10.577 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:24:10.784 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:24:10.826 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:24:10.843 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:24:10.843 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:24:10.844 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:24:10.886 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:24:11.556 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:26:11.607 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:26:11.653 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:26:12.311 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:26:12.312 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:26:12.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:26:12.512 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:26:12.553 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:26:12.567 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:26:12.567 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:26:12.569 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:26:12.612 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:26:13.273 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:28:13.329 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:28:13.374 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:28:14.093 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:28:14.094 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:28:14.094 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:28:14.300 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:28:14.387 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:28:14.404 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:28:14.404 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:28:14.406 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:28:14.450 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:28:15.150 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:30:15.206 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:30:15.257 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:30:16.001 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:30:16.002 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:30:16.002 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:30:16.212 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:30:16.255 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:30:16.272 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:30:16.272 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:30:16.273 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:30:16.315 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:30:17.010 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:32:17.070 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:32:17.113 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:32:17.803 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:32:17.804 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:32:17.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:32:18.004 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:32:18.047 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:32:18.064 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:32:18.065 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:32:18.066 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:32:18.108 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:32:18.801 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:34:18.863 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:34:18.909 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:34:19.625 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:34:19.625 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:34:19.626 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:34:19.830 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:34:19.913 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:34:19.929 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:34:19.929 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:34:19.931 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:34:19.973 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:34:20.642 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:36:20.702 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:36:20.745 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:36:21.459 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:36:21.460 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:36:21.460 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:36:21.662 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:36:21.704 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:36:21.722 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:36:21.722 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:36:21.724 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:36:21.767 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:36:22.416 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:38:22.468 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:38:22.551 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:38:23.323 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:38:23.323 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:38:23.324 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:38:23.526 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:38:23.614 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:38:23.631 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:38:23.631 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:38:23.633 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:38:23.674 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:38:24.331 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:40:24.391 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:40:24.436 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:40:25.175 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:40:25.176 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:40:25.176 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:40:25.377 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:40:25.464 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:40:25.481 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:40:25.481 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:40:25.482 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:40:25.522 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:40:26.217 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:42:26.295 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:42:26.341 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:42:27.128 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:42:27.129 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:42:27.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:42:27.333 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:42:27.377 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:42:27.436 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:42:27.437 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:42:27.438 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:42:27.478 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:42:35.615 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:44:40.875 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:44:40.921 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:44:41.792 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:44:41.793 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:44:41.793 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:44:41.994 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:44:42.084 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:44:42.102 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:44:42.102 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:44:42.104 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:44:42.145 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:44:42.858 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:46:42.913 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:46:42.957 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:46:43.635 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:46:43.636 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:46:43.636 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:46:43.846 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:46:43.933 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:46:43.947 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:46:43.948 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:46:43.949 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:46:43.995 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:46:44.668 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:48:44.729 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:48:44.774 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:48:45.487 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:48:45.487 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:48:45.488 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:48:45.688 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:48:45.731 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:48:45.748 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:48:45.749 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:48:45.751 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:48:45.793 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:48:46.506 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:50:46.569 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:50:46.614 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:50:47.306 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:50:47.307 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:50:47.307 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:50:47.515 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:50:47.607 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:50:47.626 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:50:47.626 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:50:47.628 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:50:47.670 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:50:48.347 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:52:48.403 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:52:48.450 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:52:49.172 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:52:49.173 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:52:49.173 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:52:49.373 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:52:49.461 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:52:49.480 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:52:49.480 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:52:49.481 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:52:49.524 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:52:50.253 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:54:50.305 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:54:50.349 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:54:51.044 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:54:51.045 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:54:51.045 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:54:51.251 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:54:51.339 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:54:51.356 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:54:51.356 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:54:51.358 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:54:51.400 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:54:52.063 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:56:52.117 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:56:52.163 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:56:52.908 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:56:52.909 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:56:52.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:56:53.108 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:56:53.203 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:56:53.220 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:56:53.220 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:56:53.222 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:56:53.264 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:56:53.948 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:58:54.008 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:58:54.058 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:58:54.859 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T03:58:54.859 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:58:54.860 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T03:58:55.065 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:58:55.155 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:58:55.172 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:58:55.173 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:58:55.175 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:58:55.219 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T03:58:55.895 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T04:00:55.950 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T04:00:56.036 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T04:00:56.778 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T04:00:56.778 INFO:teuthology.misc:Compressing logs... 2023-01-15T04:00:56.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T04:00:56.976 INFO:teuthology.misc:Compressing logs... 2023-01-15T04:00:57.020 INFO:teuthology.misc:Compressing logs... 2023-01-15T04:00:57.036 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T04:00:57.037 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T04:00:57.038 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T04:00:57.081 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T04:00:57.751 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T04:02:57.804 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T04:02:57.848 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T04:02:58.571 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T04:02:58.572 INFO:teuthology.misc:Compressing logs... 2023-01-15T04:02:58.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T04:02:58.773 INFO:teuthology.misc:Compressing logs... 2023-01-15T04:02:58.863 INFO:teuthology.misc:Compressing logs... 2023-01-15T04:02:58.880 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T04:02:58.881 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T04:02:58.882 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T04:02:58.926 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T04:02:59.598 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T04:04:59.655 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T04:04:59.700 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T04:05:00.460 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T04:05:00.461 INFO:teuthology.misc:Compressing logs... 2023-01-15T04:05:00.461 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T04:05:00.665 INFO:teuthology.misc:Compressing logs... 2023-01-15T04:05:00.711 INFO:teuthology.misc:Compressing logs... 2023-01-15T04:05:00.728 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T04:05:00.729 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T04:05:00.730 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T04:05:00.773 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T04:05:01.470 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T04:07:01.527 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T04:07:01.573 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T04:07:02.304 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T04:07:02.304 INFO:teuthology.misc:Compressing logs... 2023-01-15T04:07:02.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T04:07:02.554 INFO:teuthology.misc:Compressing logs... 2023-01-15T04:07:02.596 INFO:teuthology.misc:Compressing logs... 2023-01-15T04:07:02.653 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T04:07:02.654 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T04:07:02.655 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T04:07:02.697 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T04:07:03.375 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T04:09:03.432 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T04:09:03.479 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T04:09:04.228 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T04:09:04.228 INFO:teuthology.misc:Compressing logs... 2023-01-15T04:09:04.229 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T04:09:04.429 INFO:teuthology.misc:Compressing logs... 2023-01-15T04:09:04.474 INFO:teuthology.misc:Compressing logs... 2023-01-15T04:09:04.489 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T04:09:04.489 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T04:09:04.491 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T04:09:04.532 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T04:09:05.221 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T04:11:05.279 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T04:11:05.325 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T04:11:05.999 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T04:11:06.000 INFO:teuthology.misc:Compressing logs... 2023-01-15T04:11:06.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T04:11:06.250 INFO:teuthology.misc:Compressing logs... 2023-01-15T04:11:06.293 INFO:teuthology.misc:Compressing logs... 2023-01-15T04:11:06.353 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T04:11:06.353 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T04:11:06.355 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T04:11:06.399 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T04:11:07.066 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T04:13:07.137 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T04:13:07.179 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T04:13:07.869 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T04:13:07.870 INFO:teuthology.misc:Compressing logs... 2023-01-15T04:13:07.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T04:13:08.076 INFO:teuthology.misc:Compressing logs... 2023-01-15T04:13:08.172 INFO:teuthology.misc:Compressing logs... 2023-01-15T04:13:08.189 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T04:13:08.189 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T04:13:08.190 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T04:13:08.231 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T04:13:08.883 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T04:15:08.938 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T04:15:08.983 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T04:15:09.755 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T05:06:09.621 INFO:teuthology.misc:Compressing logs... 2023-01-15T05:06:09.824 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T05:06:10.654 INFO:teuthology.misc:Compressing logs... 2023-01-15T05:06:10.704 INFO:teuthology.misc:Compressing logs... 2023-01-15T05:06:10.748 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T05:06:10.749 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T05:06:10.751 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T05:06:10.801 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T05:06:11.533 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T05:08:31.409 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T05:08:31.457 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T05:08:32.180 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'client.0', 'client.1', 'client.mirror', 'client.mirror_remote'] 2023-01-15T05:08:32.384 INFO:teuthology.misc:Compressing logs... 2023-01-15T05:08:32.384 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-01-15T05:08:32.588 INFO:teuthology.misc:Compressing logs... 2023-01-15T05:08:32.832 INFO:teuthology.misc:Compressing logs... 2023-01-15T05:08:32.853 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T05:08:32.854 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T05:08:32.856 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T05:08:32.902 INFO:teuthology.kill:Killing Pids: {5905} 2023-01-15T05:08:33.639 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T05:10:33.700 WARNING:teuthology.dispatcher.supervisor:Job