2023-01-15T14:08:40.252 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2023-01-15T14:08:42.608 INFO:teuthology.lock.ops:Start node 'smithi090.front.sepia.ceph.com' reimaging 2023-01-15T14:08:42.637 INFO:teuthology.lock.ops:Updating [smithi090.front.sepia.ceph.com]: reset os type and version on server 2023-01-15T14:08:42.638 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2023-01-15T14:08:42.655 INFO:teuthology.lock.ops:Node 'smithi090.front.sepia.ceph.com' reimaging is complete 2023-01-15T14:08:42.655 INFO:teuthology.lock.ops:Start node 'smithi141.front.sepia.ceph.com' reimaging 2023-01-15T14:08:42.655 INFO:teuthology.lock.ops:Updating [smithi141.front.sepia.ceph.com]: reset os type and version on server 2023-01-15T14:08:42.655 INFO:teuthology.lock.ops:Updating smithi141.front.sepia.ceph.com on lock server 2023-01-15T14:08:42.672 INFO:teuthology.lock.ops:Node 'smithi141.front.sepia.ceph.com' reimaging is complete 2023-01-15T14:08:42.800 INFO:teuthology.provision.fog.smithi090:Scheduling deploy of ubuntu 20.04 2023-01-15T14:08:42.830 INFO:teuthology.provision.fog.smithi141:Scheduling deploy of ubuntu 20.04 2023-01-15T14:08:42.996 INFO:teuthology.orchestra.console:Power off smithi090 2023-01-15T14:08:43.825 INFO:teuthology.orchestra.console:Power off smithi141 2023-01-15T14:08:52.188 INFO:teuthology.orchestra.console:Power off for smithi141 completed 2023-01-15T14:08:52.288 INFO:teuthology.orchestra.console:Power on smithi141 2023-01-15T14:08:56.269 INFO:teuthology.orchestra.console:Power off for smithi090 completed 2023-01-15T14:08:56.370 INFO:teuthology.orchestra.console:Power on smithi090 2023-01-15T14:09:00.626 INFO:teuthology.orchestra.console:Power on for smithi090 completed 2023-01-15T14:09:00.727 INFO:teuthology.provision.fog.smithi090:Waiting for deploy to finish 2023-01-15T14:09:00.834 INFO:teuthology.orchestra.console:Power on for smithi141 completed 2023-01-15T14:09:00.935 INFO:teuthology.provision.fog.smithi141:Waiting for deploy to finish 2023-01-15T14:12:51.749 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.141 2023-01-15T14:13:00.872 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.141 2023-01-15T14:13:09.956 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.141 2023-01-15T14:13:19.012 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.141 2023-01-15T14:13:28.068 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.141 2023-01-15T14:13:50.888 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90 2023-01-15T14:13:59.972 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90 2023-01-15T14:14:06.156 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-01-15T14:14:09.092 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90 2023-01-15T14:14:18.148 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90 2023-01-15T14:14:26.919 INFO:teuthology.provision.fog.smithi141:Node is ready 2023-01-15T14:14:26.926 INFO:teuthology.orchestra.run.smithi141.stdout:smithi141.front.sepia.ceph.com 2023-01-15T14:14:26.933 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141 smithi141.front.sepia.ceph.com smithi141 2023-01-15T14:14:27.144 INFO:teuthology.provision.fog.smithi141:Deploy complete! 2023-01-15T14:14:27.368 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90 2023-01-15T14:15:33.432 WARNING:teuthology.provision.fog:timed out 2023-01-15T14:15:39.654 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-01-15T14:15:40.246 INFO:teuthology.provision.fog.smithi090:Node is ready 2023-01-15T14:15:40.252 INFO:teuthology.orchestra.run.smithi090.stdout:smithi090.front.sepia.ceph.com 2023-01-15T14:15:40.305 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90 smithi090.front.sepia.ceph.com smithi090 2023-01-15T14:15:40.500 INFO:teuthology.provision.fog.smithi090:Deploy complete! 2023-01-15T14:15:40.766 INFO:teuthology.lock.ops:Checking smithi090.front.sepia.ceph.com 2023-01-15T14:15:40.766 INFO:teuthology.lock.ops:New key found. Updating... 2023-01-15T14:15:40.793 INFO:teuthology.lock.ops:Checking smithi141.front.sepia.ceph.com 2023-01-15T14:15:40.794 INFO:teuthology.lock.ops:New key found. Updating... 2023-01-15T14:15:40.809 INFO:teuthology.lock.ops:Updating [smithi090.front.sepia.ceph.com]: set os type and version on server 2023-01-15T14:15:40.809 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-15T14:15:41.029 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2023-01-15T14:15:41.036 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="Ubuntu" 2023-01-15T14:15:41.036 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-01-15T14:15:41.036 INFO:teuthology.orchestra.run.smithi090.stdout:ID=ubuntu 2023-01-15T14:15:41.036 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE=debian 2023-01-15T14:15:41.036 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-01-15T14:15:41.036 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="20.04" 2023-01-15T14:15:41.036 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://www.ubuntu.com/" 2023-01-15T14:15:41.037 INFO:teuthology.orchestra.run.smithi090.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-01-15T14:15:41.037 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-01-15T14:15:41.037 INFO:teuthology.orchestra.run.smithi090.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-01-15T14:15:41.037 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_CODENAME=focal 2023-01-15T14:15:41.037 INFO:teuthology.orchestra.run.smithi090.stdout:UBUNTU_CODENAME=focal 2023-01-15T14:15:41.038 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2023-01-15T14:15:41.056 INFO:teuthology.lock.ops:Updating [smithi141.front.sepia.ceph.com]: set os type and version on server 2023-01-15T14:15:41.057 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-15T14:15:41.312 INFO:teuthology.orchestra.run.smithi141.stdout:x86_64 2023-01-15T14:15:41.319 INFO:teuthology.orchestra.run.smithi141.stdout:NAME="Ubuntu" 2023-01-15T14:15:41.320 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-01-15T14:15:41.320 INFO:teuthology.orchestra.run.smithi141.stdout:ID=ubuntu 2023-01-15T14:15:41.320 INFO:teuthology.orchestra.run.smithi141.stdout:ID_LIKE=debian 2023-01-15T14:15:41.320 INFO:teuthology.orchestra.run.smithi141.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-01-15T14:15:41.320 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION_ID="20.04" 2023-01-15T14:15:41.321 INFO:teuthology.orchestra.run.smithi141.stdout:HOME_URL="https://www.ubuntu.com/" 2023-01-15T14:15:41.321 INFO:teuthology.orchestra.run.smithi141.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-01-15T14:15:41.321 INFO:teuthology.orchestra.run.smithi141.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-01-15T14:15:41.321 INFO:teuthology.orchestra.run.smithi141.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-01-15T14:15:41.321 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION_CODENAME=focal 2023-01-15T14:15:41.322 INFO:teuthology.orchestra.run.smithi141.stdout:UBUNTU_CODENAME=focal 2023-01-15T14:15:41.322 INFO:teuthology.lock.ops:Updating smithi141.front.sepia.ceph.com on lock server 2023-01-15T14:15:41.406 INFO:teuthology.dispatcher.supervisor:Running job 7137228 2023-01-15T14:15:41.406 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/bin/teuthology -v --owner scheduled_rfriedma@teuthology --archive /home/teuthworker/archive/rfriedma-2023-01-12_16:28:35-rados:thrash-wip-rf-newsched-v3-distro-default-smithi/7137228 --name rfriedma-2023-01-12_16:28:35-rados:thrash-wip-rf-newsched-v3-distro-default-smithi --description rados:thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/pool-snaps-few-objects} -- /home/teuthworker/archive/rfriedma-2023-01-12_16:28:35-rados:thrash-wip-rf-newsched-v3-distro-default-smithi/7137228/orig.config.yaml 2023-01-15T14:15:41.410 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rfriedma-2023-01-12_16:28:35-rados:thrash-wip-rf-newsched-v3-distro-default-smithi/7137228 2023-01-15T14:15:41.411 INFO:teuthology.dispatcher.supervisor:Job PID: 2826 2023-01-15T14:15:41.411 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-01-16T02:17:19.724 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:17:19.867 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:17:20.602 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:17:20.602 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:17:20.602 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:17:20.602 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T02:17:20.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T02:17:21.139 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:17:21.278 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:17:30.246 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping 2023-01-16T02:17:44.173 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/ceph-mon.a.log: file size changed while zipping 2023-01-16T02:17:48.372 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/ceph-mgr.y.log: file size changed while zipping 2023-01-16T02:17:57.354 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/ceph-mon.c.log: file size changed while zipping 2023-01-16T02:18:07.519 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/ceph-osd.5.log: file size changed while zipping 2023-01-16T02:18:07.863 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/ceph-mgr.x.log: file size changed while zipping 2023-01-16T02:18:41.017 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/ceph-osd.6.log: file size changed while zipping 2023-01-16T02:18:44.623 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/ceph-osd.1.log: file size changed while zipping 2023-01-16T02:19:24.397 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/ceph-osd.4.log: file size changed while zipping 2023-01-16T02:19:33.328 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/ceph-osd.0.log: file size changed while zipping 2023-01-16T02:20:05.871 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/ceph-osd.3.log: file size changed while zipping 2023-01-16T02:20:13.501 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/ceph-osd.7.log: file size changed while zipping 2023-01-16T02:20:35.335 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/ceph-osd.2.log: file size changed while zipping 2023-01-16T02:22:03.621 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:22:04.352 INFO:teuthology.kill:Nuking machines: ['smithi090', 'smithi141'] 2023-01-16T02:22:04.997 INFO:teuthology.kill:2023-01-16 02:22:04,997.997 INFO:teuthology.nuke:targets: 2023-01-16T02:22:04.998 INFO:teuthology.kill: smithi090.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= 2023-01-16T02:22:04.998 INFO:teuthology.kill: smithi141.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0= 2023-01-16T02:22:04.998 INFO:teuthology.kill:2023-01-16 02:22:04,997.997 INFO:teuthology.nuke:Checking targets against current locks 2023-01-16T02:22:05.057 INFO:teuthology.kill:2023-01-16 02:22:05,056.056 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-16T02:22:05.087 INFO:teuthology.kill:2023-01-16 02:22:05,086.086 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-16T02:22:05.140 INFO:teuthology.kill:2023-01-16 02:22:05,140.140 INFO:teuthology.orchestra.console:Power off smithi090 2023-01-16T02:22:05.177 INFO:teuthology.kill:2023-01-16 02:22:05,177.177 INFO:teuthology.orchestra.console:Power off smithi141 2023-01-16T02:22:13.516 INFO:teuthology.kill:2023-01-16 02:22:13,516.516 INFO:teuthology.orchestra.console:Power off for smithi090 completed 2023-01-16T02:22:13.679 INFO:teuthology.kill:2023-01-16 02:22:13,679.679 INFO:teuthology.lock.ops:unlocked: smithi090.front.sepia.ceph.com 2023-01-16T02:22:17.667 INFO:teuthology.kill:2023-01-16 02:22:17,667.667 INFO:teuthology.orchestra.console:Power off for smithi141 completed 2023-01-16T02:22:17.809 INFO:teuthology.kill:2023-01-16 02:22:17,808.808 INFO:teuthology.lock.ops:unlocked: smithi141.front.sepia.ceph.com 2023-01-16T02:24:18.037 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:24:18.089 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:24:18.811 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:24:18.811 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:24:18.812 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:24:18.812 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T02:24:19.849 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi090 2023-01-16T02:24:19.900 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:24:20.615 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:26:20.773 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:26:20.824 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:26:21.619 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:26:21.619 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:26:21.619 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:26:21.619 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T02:26:24.680 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi090 2023-01-16T02:26:24.730 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:26:25.487 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:28:25.629 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:28:25.679 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:28:26.436 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:28:26.436 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:28:26.436 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:28:26.436 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T02:28:26.576 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T02:28:26.603 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:28:26.770 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:28:26.819 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:28:26.819 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:28:26.868 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:28:26.868 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:28:26.870 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:28:26.897 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2023-01-16T02:28:26.899 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2023-01-16T02:28:26.969 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T02:28:26.969 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:28:26.996 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T02:28:26.996 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:28:26.998 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:28:27.038 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T02:28:27.064 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T02:28:27.105 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:28:27.105 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:28:27.132 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:28:27.132 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:28:27.182 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:28:27.887 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:30:28.036 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:30:28.084 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:30:28.808 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:30:28.808 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:30:28.808 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:30:28.808 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T02:31:03.333 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi090 2023-01-16T02:31:03.380 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:31:04.068 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:33:04.220 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:33:04.268 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:33:05.025 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:33:05.026 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:33:05.026 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:33:05.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T02:33:05.234 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T02:33:05.282 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:33:05.481 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:33:05.533 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:33:05.533 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:33:05.586 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:33:05.586 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:33:05.588 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:33:05.820 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:33:05.861 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T02:33:05.868 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T02:33:05.948 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:33:05.949 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:33:05.992 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:33:05.992 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:33:06.042 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:33:06.726 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:35:06.878 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:35:06.928 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:35:07.681 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:35:07.682 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:35:07.682 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:35:07.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T02:35:07.890 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T02:35:07.929 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:35:08.141 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:35:08.185 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:35:08.186 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:35:08.230 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:35:08.230 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:35:08.233 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:35:08.432 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:35:08.473 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T02:35:08.505 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T02:35:08.557 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:35:08.557 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:35:08.599 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:35:08.600 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:35:08.684 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:35:09.408 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:37:09.537 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:37:09.587 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:37:10.352 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:37:10.353 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:37:10.353 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:37:10.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T02:37:10.566 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T02:37:10.605 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:37:10.820 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:37:10.864 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:37:10.864 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:37:10.907 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:37:10.908 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:37:10.910 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:37:11.129 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:37:11.170 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T02:37:11.192 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T02:37:11.254 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:37:11.254 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:37:11.300 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T02:37:11.300 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:37:11.352 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:37:12.083 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:39:12.229 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:39:12.280 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:39:13.044 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:39:13.044 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:39:13.044 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:39:13.045 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T02:39:13.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T02:39:13.298 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:39:13.514 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:39:13.559 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:39:13.559 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:39:13.601 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:39:13.602 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:39:13.603 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:39:13.646 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T02:39:13.649 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T02:39:13.706 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:39:14.555 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:41:14.696 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:41:14.747 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:41:15.490 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:41:15.490 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:41:15.490 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:41:15.491 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T02:41:15.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T02:41:15.697 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:41:15.899 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T02:41:15.945 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:41:15.946 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:41:15.993 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T02:41:15.993 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T02:41:15.994 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:41:16.038 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T02:41:16.041 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T02:41:16.090 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:41:16.839 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:43:16.991 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:43:17.045 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:43:17.792 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:43:17.793 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:43:17.793 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:43:17.793 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T02:43:17.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T02:43:18.336 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:43:18.378 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T02:43:18.381 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T02:43:18.430 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:43:19.108 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:45:19.251 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:45:19.302 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:45:20.051 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:45:20.051 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:45:20.052 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:45:20.052 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T02:45:20.267 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T02:45:20.633 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:45:20.676 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T02:45:20.679 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T02:45:20.728 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:45:21.456 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:47:21.613 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:47:21.664 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:47:22.422 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:47:22.423 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:47:22.423 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:47:22.423 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T02:47:22.636 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T02:47:23.013 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:47:23.059 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T02:47:23.061 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T02:47:23.112 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:47:23.835 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:49:23.985 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:49:24.037 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:49:24.772 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:49:24.772 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:49:24.773 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:49:24.773 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T02:49:24.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T02:49:25.336 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:49:25.382 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T02:49:25.384 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T02:49:25.457 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:49:26.173 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:51:26.348 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:51:26.405 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:51:27.156 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:51:27.156 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:51:27.156 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:51:27.156 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T02:51:27.818 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T02:51:28.178 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:51:28.231 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T02:51:28.235 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T02:51:28.284 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:51:29.041 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:53:29.207 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:53:29.255 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:53:29.980 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:53:29.980 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:53:29.980 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:53:29.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T02:53:30.203 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T02:53:30.573 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:53:30.619 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T02:53:30.621 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T02:53:30.671 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:53:31.388 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:55:31.468 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:55:31.515 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:55:32.241 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:55:32.242 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:55:32.242 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:55:32.242 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T02:55:32.450 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T02:55:32.789 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:55:32.833 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T02:55:32.835 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T02:55:32.912 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:55:33.685 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:57:33.833 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:57:33.894 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:57:34.614 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:57:34.615 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:57:34.615 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:57:34.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T02:57:34.837 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T02:57:35.172 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:57:35.218 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T02:57:35.221 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T02:57:35.272 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:57:35.974 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T02:59:36.121 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T02:59:36.167 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:59:36.941 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T02:59:36.941 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T02:59:36.941 INFO:teuthology.misc:Compressing logs... 2023-01-16T02:59:36.941 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T02:59:55.429 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi090 2023-01-16T02:59:55.480 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T02:59:56.200 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:01:56.345 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:01:56.395 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:01:57.138 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:01:57.138 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:01:57.138 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:01:57.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:02:12.506 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T03:02:13.624 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T03:02:14.744 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T03:02:15.824 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T03:02:16.943 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T03:02:18.060 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T03:02:19.137 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T03:02:20.254 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T03:02:21.373 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T03:02:22.450 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T03:02:22.451 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi090 2023-01-16T03:02:22.527 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:02:23.275 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:04:23.323 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:04:23.376 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:04:24.162 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:04:24.163 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:04:24.163 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:04:24.163 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:04:25.988 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi090 2023-01-16T03:04:26.038 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:04:26.794 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:06:26.845 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:06:26.895 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:06:27.661 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:06:27.662 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:06:27.662 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:06:27.662 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:06:27.794 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:06:27.818 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T03:06:27.992 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T03:06:28.168 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:06:28.168 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:06:28.205 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:06:28.206 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:06:28.207 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:06:28.232 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2023-01-16T03:06:28.273 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2023-01-16T03:06:28.300 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:06:28.300 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:06:28.340 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T03:06:28.340 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:06:28.341 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:06:28.366 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T03:06:28.408 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T03:06:28.435 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T03:06:28.436 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:06:28.476 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T03:06:28.476 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:06:28.531 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:06:29.308 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:08:29.376 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:08:29.429 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:08:30.207 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:08:30.207 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:08:30.208 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:08:30.208 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:08:33.288 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi090 2023-01-16T03:08:33.337 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:08:34.045 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:10:34.091 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:10:34.141 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:10:34.888 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:10:34.889 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:10:34.889 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:10:34.889 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:10:35.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:10:35.147 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T03:10:35.308 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T03:10:35.358 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:10:35.358 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:10:35.411 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:10:35.412 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:10:35.414 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:10:35.589 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:10:35.631 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T03:10:35.647 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T03:10:35.714 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T03:10:35.714 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:10:35.757 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T03:10:35.757 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:10:35.816 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:10:36.496 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:12:36.545 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:12:36.596 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:12:37.320 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:12:37.321 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:12:37.321 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:12:37.321 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:12:37.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:12:37.576 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T03:12:37.775 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T03:12:37.818 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:12:37.818 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:12:37.859 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:12:37.860 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:12:37.862 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:12:38.065 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:12:38.105 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T03:12:38.131 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T03:12:38.189 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T03:12:38.189 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:12:38.231 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T03:12:38.231 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:12:38.282 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:12:38.975 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:14:39.025 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:14:39.096 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:14:39.844 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:14:39.845 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:14:39.845 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:14:39.845 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:14:40.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:14:40.093 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T03:14:40.304 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T03:14:40.348 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:14:40.348 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:14:40.389 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:14:40.390 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:14:40.392 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:14:40.604 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:14:40.645 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T03:14:40.665 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T03:14:40.728 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T03:14:40.729 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:14:40.770 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T03:14:40.770 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:14:40.821 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:14:41.550 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:16:41.600 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:16:41.652 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:16:42.380 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:16:42.380 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:16:42.380 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:16:42.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:16:42.610 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:16:42.655 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T03:16:42.869 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T03:16:42.912 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:16:42.913 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:16:42.996 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:16:42.996 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:16:42.998 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:16:43.128 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T03:16:43.130 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:16:43.183 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:16:43.915 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:18:43.975 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:18:44.026 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:18:44.774 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:18:44.774 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:18:44.774 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:18:44.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:18:44.981 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:18:45.021 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T03:18:45.224 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T03:18:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:18:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:18:45.310 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:18:45.310 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:18:45.312 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:18:45.355 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T03:18:45.357 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:18:45.408 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:18:46.118 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:20:46.175 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:20:46.224 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:20:46.948 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:20:46.948 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:20:46.948 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:20:46.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:20:47.159 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:20:47.200 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T03:20:47.415 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T03:20:47.459 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:20:47.460 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:20:47.502 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:20:47.502 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:20:47.504 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:20:47.546 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T03:20:47.549 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:20:47.597 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:20:48.276 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:22:48.327 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:22:48.376 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:22:49.107 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:22:49.108 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:22:49.108 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:22:49.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:22:49.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:22:49.337 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T03:22:49.549 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T03:22:49.593 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:22:49.593 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:22:49.634 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T03:22:49.635 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T03:22:49.636 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:22:49.679 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T03:22:49.681 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:22:49.733 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:22:50.402 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:24:50.450 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:24:50.498 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:24:51.187 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:24:51.187 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:24:51.187 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:24:51.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:24:51.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:24:51.772 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:24:51.815 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T03:24:51.817 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:24:51.887 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:24:52.570 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:26:52.629 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:26:52.677 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:26:53.458 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:26:53.459 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:26:53.459 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:26:53.459 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:26:53.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:26:53.995 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:26:54.038 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T03:26:54.040 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:26:54.087 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:26:54.783 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:28:54.831 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:28:54.877 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:28:55.678 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:28:55.678 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:28:55.678 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:28:55.679 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:28:55.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:28:56.224 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:28:56.266 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T03:28:56.269 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:28:56.315 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:28:57.034 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:30:57.085 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:30:57.133 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:30:57.866 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:30:57.866 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:30:57.866 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:30:57.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:30:58.077 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:30:58.441 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:30:58.490 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T03:30:58.492 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:30:58.539 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:30:59.249 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:32:59.299 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:32:59.349 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:33:00.124 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:33:00.125 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:33:00.125 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:33:00.125 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:33:00.332 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:33:00.688 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:33:00.731 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T03:33:00.734 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:33:00.781 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:33:01.526 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:35:01.578 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:35:01.629 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:35:02.363 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:35:02.364 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:35:02.364 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:35:02.364 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:35:02.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:35:02.919 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:35:02.962 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T03:35:02.964 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:35:03.015 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:35:03.729 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:37:03.776 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:37:03.843 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:37:04.626 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:37:04.626 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:37:04.626 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:37:04.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:37:04.796 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:37:05.133 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:37:05.179 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T03:37:05.182 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:37:05.230 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:37:05.944 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:39:06.002 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:39:06.051 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:39:06.777 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:39:06.777 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:39:06.777 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:39:06.777 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:39:06.990 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:39:07.331 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:39:07.376 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T03:39:07.379 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:39:07.428 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:39:08.129 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:41:08.188 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:41:08.236 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:41:08.933 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:41:08.934 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:41:08.934 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:41:08.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:41:09.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:41:09.487 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:41:09.532 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T03:41:09.535 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:41:09.583 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:41:10.290 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:43:10.339 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:43:10.390 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:43:11.133 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:43:11.133 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:43:11.134 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:43:11.134 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:43:11.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:43:11.705 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:43:11.749 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T03:43:11.751 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:43:11.800 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:43:12.474 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:45:12.524 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:45:12.571 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:45:13.265 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:45:13.265 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:45:13.265 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:45:13.265 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:45:13.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:45:13.802 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:45:13.845 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T03:45:13.848 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:45:13.898 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:45:14.644 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:47:14.696 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:47:14.743 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:47:15.507 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:47:15.508 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:47:15.508 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:47:15.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:47:15.719 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:47:16.052 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:47:16.095 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T03:47:16.097 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:47:16.169 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:47:16.887 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:49:16.936 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:49:16.989 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:49:17.707 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:49:17.708 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:49:17.708 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:49:17.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:49:17.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:49:18.259 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:49:18.302 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T03:49:18.304 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T03:49:18.305 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:49:18.358 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:49:19.056 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:51:19.114 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:51:19.162 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:51:19.915 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:51:19.916 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:51:19.916 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:51:19.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:51:20.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:51:20.464 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:51:20.507 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T03:51:20.510 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:51:20.557 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:51:21.285 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:53:21.339 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:53:21.389 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:53:22.166 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:53:22.166 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:53:22.166 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:53:22.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:53:22.377 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:53:22.722 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:53:22.765 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T03:53:22.767 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:53:22.815 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:53:23.516 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:55:23.562 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:55:23.612 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:55:24.368 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:55:24.368 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:55:24.368 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:55:24.369 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:55:24.581 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:55:24.925 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:55:24.968 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T03:55:24.971 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:55:25.032 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:55:25.790 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:57:25.845 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:57:25.893 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:57:26.681 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:57:26.682 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:57:26.682 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:57:26.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:57:26.881 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:57:27.227 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:57:27.272 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T03:57:27.275 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:57:27.325 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:57:28.022 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T03:59:28.075 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T03:59:28.122 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:59:28.869 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T03:59:28.869 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T03:59:28.869 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:59:28.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T03:59:29.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T03:59:29.410 INFO:teuthology.misc:Compressing logs... 2023-01-16T03:59:29.456 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T03:59:29.458 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T03:59:29.507 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T03:59:30.275 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:01:30.324 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:01:30.375 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:01:31.173 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:01:31.174 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:01:31.174 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:01:31.174 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:01:31.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:01:31.752 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:01:31.795 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T04:01:31.797 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:01:31.846 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:01:32.588 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:03:32.645 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:03:32.696 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:03:33.434 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:03:33.434 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:03:33.435 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:03:33.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:03:33.641 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:03:33.982 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:03:34.027 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T04:03:34.029 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:03:34.083 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:03:34.789 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:05:34.849 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:05:34.897 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:05:35.642 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:05:35.642 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:05:35.642 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:05:35.642 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:05:35.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:05:36.146 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:05:36.189 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T04:05:36.192 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:05:36.240 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:05:36.911 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:07:36.959 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:07:37.011 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:07:37.771 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:07:37.772 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:07:37.772 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:07:37.772 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:07:37.984 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:07:38.325 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:07:38.369 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T04:07:38.371 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:07:38.420 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:07:39.105 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:09:39.155 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:09:39.207 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:09:39.975 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:09:39.976 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:09:39.976 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:09:39.976 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:09:40.189 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:09:40.536 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:09:40.581 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T04:09:40.581 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T04:09:40.583 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:09:40.633 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:09:41.434 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:11:41.488 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:11:41.570 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:11:42.337 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:11:42.337 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:11:42.337 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:11:42.337 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:11:42.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:11:42.847 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:11:42.891 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T04:11:42.891 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T04:11:42.893 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:11:42.943 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:11:43.634 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:13:43.683 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:13:43.735 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:13:44.463 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:13:44.463 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:13:44.463 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:13:44.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:13:44.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:13:45.024 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:13:45.069 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T04:13:45.069 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T04:13:45.071 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:13:45.133 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:13:45.846 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:15:45.903 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:15:45.954 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:15:46.712 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:15:46.712 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:15:46.712 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:15:46.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:15:46.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:15:47.250 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:15:47.293 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T04:15:47.294 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T04:15:47.295 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:15:47.345 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:15:48.048 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:17:48.101 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:17:48.149 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:17:48.872 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:17:48.873 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:17:48.873 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:17:48.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:17:49.073 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:17:49.413 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:17:49.458 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T04:17:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T04:17:49.461 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:17:49.511 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:17:50.210 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:19:50.264 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:19:50.314 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:19:51.037 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:19:51.037 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:19:51.037 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:19:51.037 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:19:51.248 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:19:51.597 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:19:51.641 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T04:19:51.641 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T04:19:51.643 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:19:51.692 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:19:52.425 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:21:52.489 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:21:52.543 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:21:53.288 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:21:53.288 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:21:53.288 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:21:53.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:21:53.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:21:53.846 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:21:53.890 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T04:21:53.891 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T04:21:53.893 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:21:53.943 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:21:54.702 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:23:54.764 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:23:54.814 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:23:55.603 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:23:55.603 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:23:55.603 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:23:55.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:23:55.813 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:23:56.146 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:23:56.189 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T04:23:56.189 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T04:23:56.193 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:23:56.244 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:23:56.972 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:25:57.030 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:25:57.088 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:25:57.846 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:25:57.846 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:25:57.846 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:25:57.846 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:25:58.062 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:25:58.400 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:25:58.443 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T04:25:58.443 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T04:25:58.445 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:25:58.492 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:25:59.219 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:27:59.266 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:27:59.318 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:28:00.044 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:28:00.045 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:28:00.045 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:28:00.045 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:28:00.245 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:28:00.592 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:28:00.635 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T04:28:00.635 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T04:28:00.638 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:28:00.691 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:28:01.422 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:30:01.474 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:30:01.525 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:30:02.305 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:30:02.306 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:30:02.306 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:30:02.306 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:30:02.517 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:30:02.856 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:30:02.899 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T04:30:02.900 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T04:30:02.901 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:30:02.951 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:30:03.693 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:32:03.769 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:32:03.826 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:32:04.629 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:32:04.629 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:32:04.629 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:32:04.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:32:04.793 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:32:05.140 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:32:05.184 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T04:32:05.184 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T04:32:05.186 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:32:05.237 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:32:05.974 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:34:06.027 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:34:06.103 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:34:06.909 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:34:06.910 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:34:06.910 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:34:06.910 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:34:07.109 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:34:07.450 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:34:07.494 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T04:34:07.494 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T04:34:07.496 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:34:07.545 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:34:08.276 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:36:08.334 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:36:08.388 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:36:09.134 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:36:09.134 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:36:09.134 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:36:09.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:36:09.337 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:36:09.677 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:36:09.721 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T04:36:09.721 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T04:36:09.723 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:36:09.773 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:36:10.544 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:38:10.603 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:38:10.653 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:38:11.412 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:38:11.412 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:38:11.413 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:38:11.413 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:38:11.621 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:38:11.968 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:38:12.014 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T04:38:12.014 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T04:38:12.016 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:38:12.065 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:38:12.788 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:40:12.855 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:40:12.902 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:40:13.614 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:40:13.614 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:40:13.615 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:40:13.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:40:13.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:40:14.172 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:40:14.215 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T04:40:14.216 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T04:40:14.218 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:40:14.273 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:40:14.981 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:42:15.029 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:42:15.077 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:42:15.856 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:42:15.856 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:42:15.856 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:42:15.857 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:42:16.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:42:16.409 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:42:16.452 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T04:42:16.453 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T04:42:16.455 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:42:16.504 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:42:17.229 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:44:17.284 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:44:17.333 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:44:18.056 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:44:18.056 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:44:18.056 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:44:18.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:44:18.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:44:18.615 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:44:18.658 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T04:44:18.658 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T04:44:18.660 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:44:18.708 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:44:19.375 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:46:19.421 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:46:19.471 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:46:20.183 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:46:20.184 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:46:20.184 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:46:20.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:46:20.387 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:46:21.071 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:46:21.117 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T04:46:21.117 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T04:46:21.120 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T04:46:21.167 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:46:21.885 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:48:22.044 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:48:22.094 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:48:22.799 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:48:22.799 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:48:22.800 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:48:22.800 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:48:23.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:48:23.378 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:48:23.420 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2023-01-16T04:48:23.421 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2023-01-16T04:48:23.504 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:48:23.504 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:48:23.545 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:48:23.545 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:48:23.547 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:48:24.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:48:24.865 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 349, in transfer_archives archive_logs(ctx, log_path, log_type) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1367, in archive_logs pull_directory(remote, remote_path, os.path.join(sub, log_path)) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 664, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi141 2023-01-16T04:48:24.913 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:48:25.594 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:50:25.650 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:50:25.699 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:50:26.400 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:50:26.401 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:50:26.401 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:50:26.401 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:50:44.773 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi090 2023-01-16T04:50:44.821 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:50:45.516 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:52:45.567 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:52:45.617 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:52:46.337 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:52:46.337 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:52:46.337 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:52:46.337 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:52:46.452 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T04:52:47.524 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T04:52:48.635 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T04:52:49.706 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T04:52:50.817 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T04:52:51.928 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T04:52:52.999 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T04:52:54.075 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T04:52:55.145 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T04:52:56.257 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T04:52:56.257 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi090 2023-01-16T04:52:56.309 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:52:57.011 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:54:57.059 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:54:57.137 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:54:57.886 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:54:57.886 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:54:57.886 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:54:57.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:54:58.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:54:58.097 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T04:54:58.280 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T04:54:58.320 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:54:58.320 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:54:58.360 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:54:58.360 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:54:58.362 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:54:58.388 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2023-01-16T04:54:58.429 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2023-01-16T04:54:58.457 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:54:58.457 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:54:58.498 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T04:54:58.498 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:54:58.500 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:54:58.527 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T04:54:58.566 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T04:54:58.594 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T04:54:58.594 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:54:58.634 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T04:54:58.634 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:54:58.686 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:54:59.427 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:56:59.475 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:56:59.525 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:57:00.250 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:57:00.251 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:57:00.251 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:57:00.251 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:57:00.427 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:57:00.452 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T04:57:00.628 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T04:57:00.654 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:57:00.655 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:57:00.695 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:57:00.695 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:57:00.696 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:57:00.851 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:57:00.877 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T04:57:00.916 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T04:57:00.945 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T04:57:00.946 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:57:00.982 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T04:57:00.982 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:57:01.033 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:57:01.743 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T04:59:01.797 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T04:59:01.852 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:59:02.645 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T04:59:02.645 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T04:59:02.646 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:59:02.646 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T04:59:02.835 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T04:59:02.863 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T04:59:03.035 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T04:59:03.066 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:59:03.066 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:59:03.102 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T04:59:03.103 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:59:03.104 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:59:03.258 INFO:teuthology.misc:Compressing logs... 2023-01-16T04:59:03.286 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T04:59:03.322 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T04:59:03.354 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T04:59:03.355 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:59:03.390 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T04:59:03.391 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T04:59:03.444 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T04:59:04.175 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:01:04.230 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:01:04.280 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:01:05.084 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:01:05.084 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:01:05.084 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:01:05.085 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:01:05.267 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T05:01:05.291 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T05:01:05.477 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T05:01:05.504 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:01:05.504 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:01:05.544 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:01:05.544 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:01:05.546 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:01:05.572 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:01:05.588 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:01:05.636 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:01:06.385 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:03:06.442 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:03:06.496 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:03:07.204 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:03:07.205 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:03:07.205 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:03:07.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:03:07.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T05:03:07.406 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T05:03:07.543 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T05:03:07.570 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:03:07.570 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:03:07.610 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:03:07.610 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:03:07.611 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:03:07.638 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:03:07.654 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:03:07.704 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:03:08.437 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:05:08.489 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:05:08.545 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:05:09.316 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:05:09.413 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:05:09.413 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:05:09.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:05:09.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T05:05:09.864 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:05:10.117 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:05:10.213 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:05:10.213 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:05:10.262 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:05:11.053 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:07:11.407 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:07:11.455 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:07:12.189 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:07:12.189 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:07:12.189 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:07:12.189 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:07:12.392 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T05:07:12.919 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:07:12.946 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:07:12.947 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:07:12.996 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:07:13.696 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:09:13.748 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:09:13.800 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:09:14.529 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:09:14.529 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:09:14.530 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:09:14.530 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:09:14.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T05:09:14.941 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:09:14.969 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:09:14.984 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:09:15.035 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:09:15.731 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:11:15.783 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:11:15.832 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:11:16.592 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:11:16.592 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:11:16.592 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:11:16.593 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:11:16.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T05:11:17.124 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:11:17.170 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:11:17.172 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:11:17.223 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:11:17.936 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:13:18.015 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:13:18.077 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:13:18.840 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:13:18.841 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:13:18.841 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:13:18.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:13:19.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T05:13:19.315 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:13:19.342 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:13:19.362 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:13:19.412 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:13:20.148 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:15:20.203 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:15:20.282 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:15:28.213 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:15:28.214 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:15:28.214 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:15:28.214 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:15:28.389 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T05:15:28.670 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:15:28.738 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:15:28.769 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:15:28.771 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:15:28.820 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:15:29.600 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:17:29.657 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:17:29.709 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:17:30.437 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:17:30.437 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:17:30.437 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:17:30.438 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:17:30.796 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T05:17:31.248 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:17:31.288 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:17:31.337 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:17:31.340 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:17:31.389 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:17:32.156 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:19:32.214 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:19:32.267 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:19:35.249 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:19:35.249 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:19:35.249 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:19:35.250 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:19:35.762 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T05:19:37.112 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:19:37.175 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:19:37.387 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:19:37.388 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T05:19:37.390 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:19:37.440 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:19:38.156 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:21:38.206 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:21:38.255 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:21:44.093 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:21:44.094 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:21:44.094 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:21:44.094 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:21:44.274 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T05:21:44.553 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:21:44.582 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:21:44.582 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T05:21:44.595 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:21:44.645 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:21:45.333 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:23:45.388 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:23:45.439 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:23:46.181 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:23:46.181 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:23:46.181 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:23:46.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:23:46.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T05:23:46.684 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:23:46.713 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:23:46.714 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T05:23:46.726 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:23:46.776 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:23:47.573 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:25:47.621 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:25:47.676 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:25:48.401 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:25:48.401 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:25:48.401 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:25:48.401 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:25:48.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T05:25:49.766 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:25:49.905 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:25:50.087 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:25:50.087 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T05:25:50.089 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:25:50.142 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:25:50.844 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:27:50.895 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:27:50.947 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:27:51.694 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:27:51.694 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:27:51.694 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:27:51.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:27:51.893 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T05:27:52.965 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:27:53.066 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:27:53.066 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T05:27:53.078 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:27:53.127 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:27:53.855 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:29:53.905 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:29:53.955 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:29:54.709 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:29:54.709 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:29:54.710 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:29:54.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:29:54.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T05:29:55.659 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:29:55.687 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:29:55.687 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T05:29:55.688 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:29:55.689 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:29:55.749 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:29:56.483 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:31:56.536 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:31:56.585 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:31:57.290 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:31:57.290 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:31:57.290 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:31:57.290 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:31:57.474 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T05:31:58.082 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:31:58.119 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:31:58.119 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T05:31:58.124 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:31:58.172 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:31:58.869 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:33:58.922 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:33:58.973 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:33:59.726 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:33:59.727 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:33:59.727 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:33:59.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:33:59.905 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T05:34:00.327 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:34:00.356 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:34:00.356 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T05:34:00.369 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:34:00.421 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:34:01.173 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:36:01.232 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:36:01.277 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:36:01.986 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:36:01.987 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:36:01.987 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:36:01.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:36:02.163 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T05:36:02.435 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:36:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:36:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T05:36:02.477 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:36:02.552 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:36:03.285 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:38:03.335 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:38:03.387 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:38:04.122 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:38:04.122 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:38:04.122 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:38:04.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:38:04.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T05:38:04.585 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:38:04.620 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:38:04.620 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T05:38:04.627 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:38:04.676 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:38:05.418 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:40:05.470 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:40:05.519 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:40:06.253 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:40:06.254 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:40:06.254 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:40:06.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:40:06.387 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T05:40:06.670 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:40:06.697 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:40:06.697 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T05:40:06.712 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:40:06.763 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:40:07.466 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:42:07.515 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:42:07.568 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:42:08.316 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:42:08.316 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:42:08.316 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:42:08.316 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:42:08.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T05:42:08.783 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:42:08.810 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:42:08.810 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T05:42:08.826 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:42:08.877 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:42:09.587 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:44:09.637 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:44:09.696 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:44:10.489 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:44:10.489 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:44:10.489 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:44:10.490 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:44:10.662 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T05:44:10.961 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:44:10.996 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T05:44:10.996 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-16T05:44:11.004 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T05:44:11.066 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:44:11.874 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:46:11.927 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:46:11.973 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:46:12.686 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:46:12.686 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:46:12.686 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:46:12.686 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:47:12.688 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi090 2023-01-16T05:47:12.739 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:47:13.471 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:49:13.529 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:49:13.577 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:49:14.294 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:49:14.294 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:49:14.294 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:49:14.294 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:49:14.368 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T05:49:15.441 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T05:49:16.553 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T05:49:17.627 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T05:49:18.740 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T05:49:19.813 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T05:49:20.926 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T05:49:21.999 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T05:49:23.114 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T05:49:24.188 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T05:49:24.189 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi090 2023-01-16T05:49:24.245 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:49:24.957 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:51:25.016 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:51:25.073 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:51:25.820 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:51:25.821 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:51:25.821 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:51:25.821 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:52:25.825 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi090 2023-01-16T05:52:25.869 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:52:26.595 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:54:26.645 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:54:26.693 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:54:27.393 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:54:27.393 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:54:27.394 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:54:27.394 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:54:27.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T05:54:27.607 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T05:54:27.789 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T05:54:27.890 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:54:27.890 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:54:28.084 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:54:28.084 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:54:28.085 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:54:28.114 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2023-01-16T05:54:28.118 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2023-01-16T05:54:28.183 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:54:28.184 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:54:28.213 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T05:54:28.213 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:54:28.215 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:54:28.252 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T05:54:28.282 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T05:54:28.321 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T05:54:28.321 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:54:28.351 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T05:54:28.351 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:54:28.402 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:54:29.107 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:56:29.154 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:56:29.202 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:56:29.948 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:56:29.948 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:56:29.948 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:56:29.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:56:48.420 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi090 2023-01-16T05:56:48.476 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:56:49.174 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T05:58:49.229 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T05:58:49.281 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:58:50.062 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T05:58:50.062 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T05:58:50.062 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:58:50.062 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T05:58:50.269 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T05:58:50.317 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T05:58:50.520 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T05:58:50.576 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:58:50.576 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:58:50.629 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T05:58:50.629 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:58:50.631 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:58:50.792 INFO:teuthology.misc:Compressing logs... 2023-01-16T05:58:50.833 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T05:58:50.835 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T05:58:50.918 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T05:58:50.918 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:58:50.961 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T05:58:50.962 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T05:58:51.044 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T05:58:51.791 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:00:51.838 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:00:51.887 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:00:52.656 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:00:52.656 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:00:52.656 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:00:52.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T06:00:52.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T06:00:52.913 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T06:00:53.128 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T06:00:53.174 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:00:53.174 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:00:53.217 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:00:53.217 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:00:53.219 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:00:53.417 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:00:53.459 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T06:00:53.460 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T06:00:53.544 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T06:00:53.545 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:00:53.588 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T06:00:53.588 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:00:53.641 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:00:54.349 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:02:54.400 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:02:54.451 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:02:55.179 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:02:55.180 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:02:55.180 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:02:55.180 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T06:02:55.402 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T06:02:55.441 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T06:02:55.662 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T06:02:55.707 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:02:55.707 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:02:55.751 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:02:55.751 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:02:55.753 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:02:55.796 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T06:02:55.799 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T06:02:55.850 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:02:56.563 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:04:56.615 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:04:56.666 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:04:57.445 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:04:57.446 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:04:57.446 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:04:57.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T06:04:57.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T06:04:57.703 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T06:04:57.909 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T06:04:57.953 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:04:57.954 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:04:57.996 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:04:57.997 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:04:57.998 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:04:58.043 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T06:04:58.045 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T06:04:58.096 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:04:58.849 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:06:58.934 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:06:58.983 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:06:59.709 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:06:59.710 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:06:59.710 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:06:59.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T06:06:59.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T06:06:59.930 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T06:07:00.109 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T06:07:00.157 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:07:00.157 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:07:00.202 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:07:00.202 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:07:00.204 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:07:00.248 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T06:07:00.252 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T06:07:00.302 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:07:01.024 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:09:01.073 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:09:01.124 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:09:01.873 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:09:01.874 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:09:01.874 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:09:01.874 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T06:09:03.140 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi090 2023-01-16T06:09:03.190 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:09:03.904 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:11:03.953 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:11:04.000 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:11:04.692 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:11:04.692 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:11:04.692 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:11:04.692 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T06:11:04.765 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T06:11:05.838 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T06:11:06.950 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T06:11:08.061 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T06:11:09.174 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T06:11:10.285 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T06:11:11.396 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T06:11:12.506 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T06:11:13.616 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T06:11:14.728 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed. 2023-01-16T06:11:14.728 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi090 2023-01-16T06:11:14.776 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:11:15.438 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:13:15.489 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:13:15.543 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:13:16.322 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:13:16.323 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:13:16.323 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:13:16.323 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T06:13:48.605 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T06:13:48.742 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T06:13:48.883 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T06:13:48.963 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:13:48.963 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:13:49.000 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:13:49.000 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:13:49.001 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:13:49.027 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2023-01-16T06:13:49.068 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2023-01-16T06:13:49.095 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T06:13:49.096 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:13:49.136 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T06:13:49.136 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:13:49.137 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:13:49.163 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T06:13:49.203 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T06:13:49.231 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T06:13:49.231 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:13:49.271 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T06:13:49.271 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:13:49.321 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:13:50.009 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:15:50.063 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:15:50.119 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:15:50.837 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:15:50.837 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:15:50.837 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:15:50.837 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T06:15:51.015 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T06:15:51.039 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T06:15:51.588 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T06:15:51.616 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:15:51.616 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:15:51.656 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:15:51.656 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:15:51.657 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:15:51.682 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2023-01-16T06:15:51.723 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2023-01-16T06:15:51.750 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T06:15:51.750 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:15:51.790 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-16T06:15:51.790 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:15:51.791 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:15:51.817 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T06:15:51.857 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T06:15:51.884 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T06:15:51.884 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:15:51.925 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T06:15:51.925 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:15:51.975 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:15:52.638 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:17:52.688 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:17:52.738 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:17:53.470 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:17:53.471 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:17:53.471 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:17:53.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T06:18:03.812 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi090 2023-01-16T06:18:03.861 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:18:04.580 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:20:04.642 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:20:04.692 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:20:05.430 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:20:05.430 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:20:05.430 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:20:05.431 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T06:20:05.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T06:20:05.775 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T06:20:06.024 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T06:20:06.138 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:20:06.138 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:20:06.260 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:20:06.261 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:20:06.267 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:20:06.617 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:20:06.718 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T06:20:06.721 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T06:20:06.829 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T06:20:06.829 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:20:06.931 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T06:20:06.931 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:20:07.017 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:20:07.725 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:22:07.778 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:22:07.825 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:22:08.569 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:22:08.569 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:22:08.569 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:22:08.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T06:22:08.812 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T06:22:08.909 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T06:22:09.151 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T06:22:09.261 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:22:09.261 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:22:09.363 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:22:09.364 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:22:09.369 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:22:09.716 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:22:09.818 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T06:22:09.829 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T06:22:09.940 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T06:22:09.940 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:22:10.044 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T06:22:10.045 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:22:10.099 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:22:10.859 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:24:10.913 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:24:10.965 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:24:11.732 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:24:11.732 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:24:11.732 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:24:11.733 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T06:24:11.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T06:24:12.076 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T06:24:12.316 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T06:24:12.423 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:24:12.424 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:24:12.528 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:24:12.528 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:24:12.533 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:24:12.879 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:24:12.981 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T06:24:12.987 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T06:24:13.096 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T06:24:13.096 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:24:13.203 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T06:24:13.203 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:24:13.258 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:24:13.977 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:26:14.029 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:26:14.080 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:26:14.829 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:26:14.829 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:26:14.830 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:26:14.830 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T06:26:15.077 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T06:26:15.173 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T06:26:15.418 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T06:26:15.525 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:26:15.525 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:26:15.629 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:26:15.629 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:26:15.634 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:26:15.983 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:26:16.082 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T06:26:16.087 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-16T06:26:16.196 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T06:26:16.196 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:26:16.299 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-16T06:26:16.299 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:26:16.356 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:26:17.093 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:28:17.149 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:28:17.201 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:28:18.029 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:28:18.029 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:28:18.029 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:28:18.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T06:28:18.293 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T06:28:18.388 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T06:28:18.653 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T06:28:18.759 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:28:18.760 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:28:18.867 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:28:18.868 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:28:18.874 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:28:18.978 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T06:28:18.983 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T06:28:19.039 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:28:19.855 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:30:19.918 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:30:19.973 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:30:20.767 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:30:20.767 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:30:20.767 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:30:20.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T06:30:21.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T06:30:21.114 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T06:30:21.350 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-16T06:30:21.456 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:30:21.456 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:30:21.557 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-16T06:30:21.557 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-01-16T06:30:21.564 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:30:21.667 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T06:30:21.673 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T06:30:21.732 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:30:22.572 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:32:22.625 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:32:22.686 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:32:23.571 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-16T06:32:23.571 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-16T06:32:23.571 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:32:23.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-16T06:32:23.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-16T06:32:24.393 INFO:teuthology.misc:Compressing logs... 2023-01-16T06:32:24.503 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T06:32:24.505 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-16T06:32:24.510 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-16T06:32:24.562 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:32:25.332 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-16T06:34:25.386 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-16T06:34:25.462 INFO:teuthology.kill:Killing Pids: {2826} 2023-01-16T06:34:26.329 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a']